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INTRODUCTION 



Research for Better Schools, Inc. since its inception 
in 1966, has pressed for improvements in methods used for planning 
and decision-making in schools. Dr. Sanford Temkin has focused 
major attention upon this problem in the more than three years 
he has been a professional staff member at RBS. His doctoral 
dissertation presents a planning and decision-making method which 
holds great promise for application after more is learned about 
change itself as well as about the changes which are imposed on 
schools as they assimilate planning methods. The method presented 
in his dissertation rests firmly on three foundations: statistical 

and mathematical modeling, economic theory, and expexience in educa- 
tion and educational administration. 

Tho preeminent reason for reprinting this dissertation is 
the contribution it has made to RBS in the development of a new 
program. This new program, the Administering for Change Program, was 
formally created during the summer of 1968. One of its major goals 
is to help local school administrators make it possible to bring about 
the changes they deem necessary. 

RBS takes both pleasure and pride in presenting Dr. Temkin’ s 
work with the hope it may he as useful to others as it has been to us. 
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I. THE PROBLEM AND ITS SETTING 



A. Background 

Schools exist to provide services for children. School programs 
(e.g. arithmetic curriculum; lunch program) are the products which the 
school system produces in an effort to meet the needs of the pupil 
population. 

One of the major problems which confronts the school system is the 
problem of resource allocation since demands for resources are greater 
than the supply of resources determined by the current operating 
budget level. This presupposes that the school decision-maker^ does not 
generally have a budget of sufficient size to allow for the implementation 
of all the programs he deems necessary. Further, it implies that there 
is an opportunity cost associated with running t’e system's programs » since 
a given amount of resources expended for one program restricts the amount 
of resources available for use by others. The economic problem is to 
allocate the given level of resources to programs in such a way that the 
maximum contribution to the system's objective function is realized. 

The magnitude of the problem is considered by Leonard Lecht in a 
recent book. Lecht states that expenditures for education were in the 
neighborhood of $30 billion in 1962. Assuming increasing costs and 
intensified emphasis on educational goals, Lecht predicts that more than 
$80 billion will be required by 1975 (in 1962 dollars).^ 

^In this study reference is made interchangeably to the decision-maker 
and school superintendent. 

O 

“^Leonard A. Lecht, Goals, Priorities^ and Dollars; The Next Decade , 
(New York; The Free Press, 1966), p. 140. 
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Alexander Mood, former Assistant Commissioner of Education, referring 
to the cost of the nation's educational system, said: 

It is a very expensive element too; any contribution 

uiio u v^axi Dc tuduc; uv/ cx 

will pay handsome returns. 

This is narrowed to the specific domain of public school resource 

allocation in a study by Abt Associates. 

Millions of dollars are spent every day in the 
United States to improve the public schools, yet 
the quantity and quality of education available 
to many is believed inadequate to meet public 
demand. Since national human and physical 
resources potentially usable in the improvement of 
the schools are competed for by other national needs, 
only limited resources are available for schools. 

When improvements are desired and only limited amounts 
of the necessary resources are available the effi- 
ciency of resource allocation becomes a critical 
problem.^ 

Before the author suggests a specific direction for helping school 

systems, it is necessary to understand some of the problems school 

decision-makers face when contemplating resource allocation. One of the 

difficulties concerns public response to educational innovation. Charles 

Benson discusses this situation. 

...The public schools are local monopolies and 
hence cannot in fairness make any kind of radical 
change which would be repellent to some groups of 
parents. Second, inventions and innovations cost 
money. They are processes involving risk, in the 
sense that the 'payoff* is uncertain, with respect 
to whether any good thing will occur and, if so. 






5 



or 



0XX xcLcncy 



3 

Alexander M. Mood and R. Powers, Cost-Benefit Analysis of Education , 
Paper presented at Washington Operations Research Council's Second Cost- 
Effectiveness Symposium, (Washington: March 1967), p. 1. 

^Design For an Elementary and Secondary Education Cost-Effectiveness 
Model: Volume I Model Description , for the U.S. Office of Education 
(Cambridge: Abt Associates, June, 1967), p. 1. 
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when it will occur. It follows that expenditures 
on development, broadly considered, are hard to 
defend against the attacks of the zealous skeptic.^ 

Again, it is Benson who states that school resource allocation 
decisions tend to be made on ad hoc bases, primarily by committee, 
augmented in some instances by teachers and consultants.^ Planning of 
this type and th^ resulting decision-making generate a high degree of 
suboptimization. Suboptimization is usually taken to mean optimization 
in ignorance of higher order considerations, but it could be thought of 
in the parallel sense, e.g. a research program, trying to dewlop better 
ways to teach a one year course in trigonometry while the Mathematics 
Coordinator tries to reduce the amount of time devoted to trigonometry 
to four school weeks. 

In terms of the marketplace having an influence on the production of 
school services, we find that the consumers of these services do not 
directly purchase the amounts they desire. Consequently, their prefer- 
ences are not reflected in market prices. In addition, there is little 
reason to assume that these consumers possess the abilities to make 
rational selections from among the available educational products. The 
conclusion that educators know very little about t educational production 
function does not enhance the outlook for efficient allocation.^ These 
factors, coupled with the absence of competition, indicate that outside 
of public response and limited resources there is little pressure brought 
to bear on schools to produce educational outputs more efficiently. 

^Charles S. Benson, The Economics of Public Education , (Boston: 
Houghton-Miff lin Co., 1961), p. 3Z7. 

^Benson, op. cit . , p. 359. 

This conclusion has been drawn by numerous authors. An example is 
found in: Jesse Burkhead, Thomas, B. Fox, and John W. Holland, Input and Out- 
put in Large-City High Schools , (Syracuse: Syracuse University 



B. The Purpose and Plan of This Study 



The general economic problem is to allocate the given level of 
resources in such a way that the system* s outputs are maximized. It is 
clear, however, that any improvement in the way schools allocate their 
resources can be important. The purpose of this study is to develop a 
general model for evaluating resource allocations and apply it to the 
ongoing programs of a school system. 

The plan of this study begins in this chapter with a description 
(section C) of some of the methods employed in making public sector 
resource allocation decisions. A conclusion is drawn relative to the 
appropriateness of each method for evaluating the, ongoing programs of a 
school system. 

In the second chapter a nine-case structure is developed and analyzed 
in order to lay the groundwork for the evaluation of ongoing school systems. 
The analysis of these cases leans heavily on indifference curve theory, 
certainty-equivalence theory, and statistical theory. 

Chapter III considers the problems encountered in an application of 
the model to the Health and Physical Education Sub-System of the Radnor 
Township School District. As a case study it develops the data systems 
required by the model. The next chapter provides model outputs and 
considers the utility of these for the decision-maker. Chapter V uses 
the cost-effectiveness data derived earlier and considers the output 
implications of selected budget strategies for the pilot school sub-system. 
In addition, this chapter explores the sensitivity of model outputs to 
various changes in the decision-maker's value system. 

The final chapter provides specific recommendations to the decision- 




maker as a result of the case study and also extends a more general set 
of cautions and recommendations. Finally, suggestions for future 
research are presented. 

C. Systematic Analysis and Resource Allocation Decisions 

The problem of allocating resources to programs can be viewed in 
two ways with regard to the time dimension. If no programs exist and 
there is a budget to allocate, then the analyst provides an a^ priori 
decision framework. If, on the other hand, programs do exist, then the 
viewpoint of this dissertation is that the analyst should consider an 
a posteriori or evaluative framework. Evaluation is constrained by the 
realities of an existing system of ongoing programs andi therefore, 
programs are to be evaluated so that recommendations for subsequent 
period program modifications can be made. 

Much of the literature dealing with public sector resource 
allocation decision has tended to adopt an a priori methodology. That 
is, the authors have focused on the selection of a program(s) from a set 

O 

of alternatives, but in advance of the actual operation of any programs. 
If we were contemplating the design and development of a completely new 
school system, we would first come to agreement as to our objectives. 

Then we would consider anticipated costs and benefits associated with the 
respective programs contributing to the system objectives. This type of 
situation would lend itself to a. priori decision models. 

g 

Some examples of carefully structured general analyses are: 

Morris Hamburg, Statistical Decision Theory and Benefit-Cost Analysis for 
Preferredness of Choice Among Alternative Projects , a paper presented at 
the 12th International Meeting of the Institute of Management Sciences 
(Vienna: September, 1965). 

Arthur Smithies, Government Decision-Making and The Theory of Choice , 
(Santa Monica: The Rand Corporationi 1964). 
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However, the evaluation of ongoing school systems is different. 

Some of the reasons for this difference are those which explain the 
tendency for institutions to perpetuate themselves. In addition, even 
if school systems could be systematically designed and developed, one 
is not sure whether enough is understood about the political, social, 
and economic forces that interact to comprise the often subtle environ- 
ment within which schools perform. 

Literature devoted to resource allocation decisions in the public 
sector has been quite diverse in terms of area of application. Some 
of the fields of application have been recreation, public health, 
weapons systems, highway construction, water resource development, 
food irradiation, and education. Studies, mostly in the format of 
methodological proposals, have been referred to as benefit-cost analysis, 
cost-effectiveness analysis. Program Planning Budgeting Systems (PPBS) , 
input-output analysis, etc. The remainder of this section describes 
these models. Each description is a generalization, in that studies 
purporting to use a given method (e.g. benefit-cost analysis) often 
have little in common. When possible, the method is described as it 
is commonly applied to school systems. 

1, Benefit-Cost Analysis 

This approach derives from early work in the 1920' s, primarily 
in the field of water resource development. Methodologically it aims 
at the selection of one or more projects from a set of alternatives. 

Each is * ^wed as a capital investment and the analysis focuses on the 
benefits and costs. 

One of the initial steps involves a statement of goals or objec- 
tives and a subsequent translation into operational terms. On these 
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operational bases, dollar valuations are assigned to benefit and cost 
components. Several studies attempting to assess lifetime earnings as 
a function of various present-period programs have been conducted. 
Educators have been reluctant to embrace conclusions deriving from 
the studies partly because they see an open society. That is, children 
who are educated in the present-period are likely to migrate from the 
school district. Many immediate pressures for the school decision- 
maker, coupled with a rational lack of concern for the distant future, 
cause him to overemphasize the importance of present-period consumption 
of school services at the expense of future-oriented production 
considerations. 

Cost problems are also complex, but cost estimation is less 
complex than benefit estimation due to several factors; 1) there are 
specialists such as business managers, accountants, and cost estimators 
who have experience in working with costs, 2) historical cost data 
are often available for use as a base for projections, and 3) most costs 
can be reduced to a common denominator of dollars. 

The benofit-cost criterion has several forms. For the sake of 
discussion the formulation commonly referred to as the “present value 
of net benefits criterion" is presented below. 



Let 



Bj. = project benefits for 
year t 

= project costs for year t 




T ” the project duration 

(i.e.the number of years 
project benefits and 
costs are to be explic- 



itly included in the 
analysis) 



i = discount rate 

V * net present value of 



the project. 
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Many of the controversies and arguments arise from attempts to 
operationally define these symbols under realistic conditions. Higher 
order arguments focus on the nature of the overall criterion. 

Controversy derives from the insistence by most experts that benefit- 
cost ratios are poor criteria. When benefits can be reduced to a 
dollar unit of measurement there is little argument, but in many 
public sector instances this is not feasible. Otto Eckstein, the 
leading proponent of the ratio, explores the nature of project rankings 
by using ratios and present value of net benefits discounted by the 
internal rate of return. Eckstein's arguments and his explicit 
assumptions can be found in a lengthy effort comparing the two criteria 
as bases for ranking projects.^ 

Most discussions of overall benefit-cost criteria are quick to 
point out that there is no universal criterion. The "Green Book" 
suggests the criterion of maximization of net benefits for comparing 
different projects. The problem of the ratio is treated in detail in 
Appendix A in conjunction with an analysis of an argument between 
Roland McKean and Eckstein. 

Another argument centers on the selection of the appropriate 
discount rate. Roland McKean provides an extensive discussion of this 
topic. He indicates that under conditions of no capital rationing 

^Otto Eckstein, Water Resource Development , (Cambridge: Harvard 
University Press, 1958), pp. 55-57. 

l^This is common terminology fo r Proposed Practices for Economic 
Analysis of River Basin Projects , Report to the Federal Inter-Agency 
River Basic Committee prepared by the Sub-Committee on Benefits and Costs, 
(Washington: 1950). 

^^Roland N. McKean, Efficiency in Government Through Systems Analysis, 
(New York: John Wiley & S6nsTThc.“, T9'5H);pp. 76-92. ^ 
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(ie® borrowing for projects is pcs^rible) the agency should invest in 
projects until the next project in e?'pected to yield no more 

than the cost of borrowing (i) . The expected yiexd of a project, is 
called its internal rate of return (r)^ i.e. that rate of discount which 
makes its present value equal to zero. Therefore, investment in projects 
should continue as long as r>i and the interest rate can be viewed 
as the marginal internal rate of return. 

In the capital rationing instance, the budget could be such that 
the relationship between r and i is not relevant. Therefore, under 
conditions of scarce capital, McKean argues for the use of the marginal 
internal rate of return as the appropriate discount factor. The inter- 
nal rate of return is the yield that could be earned in the next-best 
opportunity available to the investor. 

The examples given by McKean treat such considerations as reinvest- 
ment of receipts, resale value of the investment, consumption of the 
receipts, etc. 

The benefit-cost literature abounds with other types of arguments. 
Some of these are: 

1. The selection of the project's duration of planning horizon (T 
in the formulation given previously) is usually arbitrary, but 
the ranking of projects is sensitive to this choice. 

2. There is considerable disagreement as to the choice of the 
appropriate opportunity cost (discount rate) in a given field. 

3. The use of "shadow prices" to enable the analyst to value 
benefits in dollar units due to the inapplicability of market 
prices in most public sector domains is a problem. Also related 



12john V. Krutilla and Otto Eckstein. Multiple Purpose River Develop - 
ment, Resources for the Future , Inc. , (Baltimore: The Johns Hopkins Press, 
1958), pp. 125-127. 

l^Roland N. McKean, "The Use of Shadow Prices , "Samuel B. Chase, Editor , 
Problems in Public Expe nditure Analysis , (Washington: The Brookings Insti- 
tution, lybb; , pp. 33-34^ 
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to this are the pr»oblems of treating intangibles and incommen- 
surableso 

4. The treatment of uncertainty in estimating benefits and costs, 
which is not unrelated to the choice of planning horizon, is 
troublesome and usually neglected quantitatively although 
considered generally in study conclusions. 

5. Suboptimization resulting from, attempts at optim.izing within 
components of the system so as to make the analysis manageable 
is, in general, a necessary danger. 

In summary, it is clear that benefit-cost analysis has no consistent 
format or prescription. Objectives are defined and alternatives are 
structured and analyzed in ttrms of benefits and costs. The attitude 
seems to be strictly £ priori investment, with an implication, if one 
reads between the lines, that a benefit-cost analysis generates 
sequential decisions, especially in the constrained budget case (capital 
rationing). The major advantage seems to be the explicitness demanded 
by analysis of public sector ir stments within a framework provided 
by private sector investment theory. 

The applications of this method to education have been sparse. 

Most of the applied efforts have centered on a comparison of programs 
based on the net present value of a stream of lifetime earnings. 

2. Cost-Effectiveness Analysis 

This approach derives from military jystems analysis where the 
problem is to select a system design from a set of alternatives designed 

^^or example: Theodore W. Schultz, The Economic Value of Education , 
(New York: Columbia University Press, 1963). 

Burton A. Weisbrod, "Preventing High School Dropouts," Robert 
Dorfman, Editor, Measuring Benefits of Government Investments , (Wash- 
ington: The Brookings Institution, 1963). 
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to meet one or more objectives. In this framework the analysis focuses 
on the effectiveness and cost of tho respective alternatives. It is a 
natural substitute for benefit-cost analysis for situations i.n which the 
benefits are incommensurable and inappropriate for dollar valuation. 



The situation is often viewed as one of finding a minimum cost 
outlay for a given level of effectiveness across system objectives 
(although actual applications usually deal with a single objective and 
a single measure of effectiveness) . Major problems arise out of efforts 
to have criteria to stand in proxy for measures of the extent to which 
objectives have been met. 

Some operational definitions of effectiveness are interesting. In 
an Air Force problem, the objective was to "optimize the bomber effective 
ness by trading off reliability, maintainability, performance, and cost 
factors." Here, two criteria were developed snd a term "strategic 
effectiveness" was defined as the probability of success in terms of 
meeting each criterion. 



^^Another such approach is "cost-utility" analysis, although Gene 
Fisher says 

Attempting to define cost-utility analysis poses somewhat of a 
semantics problem. Numerous terms are in current use which con- 
vey the same general maaning, but which have important different 
meanings to different people! cost-benefit analysis, cost-cffec- 
tiveness analysis, systems analysis, operations research, opera- 
tions analysis, etc. Because of such teminological confusion, in 
this chapter all of these terms are rejected and instead, *cost- 
utility analysis* is employed. 

This paragraph appears in: G. H. Fisher, The Role of Cost-Utility 
Analysis In Program Budgeting , (Santa Monica: The Rand Corpora tion, 1964) , 

p. 3. 

^ "Cost-Effectiveness Optimization . (Technical Supplement) , Final 
Report Task Group IV, Weapon System Effectiveness Industrial Advisory 
Committee, Headquarters, Air Force Systems Command, Andrews Air Force 
Base, Maryland ,( January, 1965), pp. 24"29. 
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A paper presenting a model which relates policies to costs argues 
for establishing cost-effectiveness trade-offs by simulation when effec- 



One proposal suggests that a production function be defined in 



where A is an achievement score for an individual 
X is a vector of non-school variables 
Y is a vector of non-teacher characteristics of the 
school 

is a teacher attribute 



Under certain assumptions (F is convex to the origin and continuous 
throughout its domain with positive first order partial derivatives and 
negative second order partial derivatives) and with a budget constraint, 
the author suggests the familiar equality 



as the relationship which produces the most for a given child. When 



an increase or decrease in will lose in terms of achievement for the 
individual. Although the author does not pursue the analysis to indi- 
cate implications for schools, he does attempt to measure empirically 



Simulation , Management Science Center , Wharton School of Finance and Com- 
merce , university of Pennsylvania , (Philadelphia :April 1967). 



P olicy-Profession, Confusion, Promise, Research and Development Memorandum 
NOc .41, Stanford Center for Research and Development in Teaching, 

Stanford University (Stanford: December, 1968), pp. 5-6. 



tiveness is defined as performance.^^ 



the form 



A = F(X,Y,Z, 



• • • > 




S>A/ 



= 9A/ 9Z|^ , P. is price of teacher 



the ratio of 9a/ d Zi is c..nstant for all k teacher attributes then 



some input-output values by multiple regression techniques.^® 



^^Blanning, Robert W. , Opportunity Cost and Effectiveness Analysis by 



18 

Henry M. Levin, Cost-Effectiveness Analysis and Educational 
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Another effectiveness concept is weighted effectiveness. This is 

found in a discussion of how to handle the problem of two or more 
19 

objectives. 

Interestingly, Ackoff suggests expected value as a way to define 
effectiveness in a posteriori casesc He Indicates that expected effec- 
tiveness is optimal only if the value function in relation to the scale 

. - . • • 20 
is linear and monotcnicaiiy increasing. 

There are other ways in which effectiveness has been defined , but 
applications to educational problems have been sparse. For the most 
part, analyses have focused on before-the-fact system design, and not 
after-the-fact system evaluation. Cost-effectiveness offers possibili- 
ties since it overcomes the problem of incommensurable units, provided 
another common denominator or value system can be substituted. 



3. Program Planning Budget Systems 
This approach received its principal impetus from the studies 
conducted in the Department of Defense under Secretary McNamara. The 
method attacks the resource allocation problem through the system s 
accounting-fiscal mechanisms. It is an attempt to integrate planning 
(setting objectives and policies), programming (specifying what is to be 
done to accomplish the objectives) , and budgeting (specifying intended 
allocations of resources in given intervals of time in the typical 
PPBS situation one usually encounters a five-year planning interval). 

^\ussell L. Ackoff, Shiv K. Gupta, and Sayer J. Minas, Scientific 
Method , (New York: John Wiley & Sons, Inc., 1962), pp. 76-77. 

^^Ackoff, et. al. ibid . . pp, 103-104. 
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The Intermediate Unit Planning Study of the Pels Institute of Local 
and State Government, University of Pennsylvania, is presently trying to 
develop a PPB3 model for educational administrative areas (Intermediate 
Units) in Pennsylvania. A paper by Wilson and Gupta in connection with 
this study reviews the literature on the application of systems analysis 
techniques to the managing of a school system. They indicate that 
research in this respect can be sorted in three gross categories: 

1. PPBS efforts in some school systems. 

2. Systems analysis applications to school system supportive 
functions. 

3. Attempts to describe the educational process aiming toward the 
development of predictive tools. 

They further indicate that the magnitude of the problems associated 

with the application of a PPBS model for a school system has caused some 

school systems to terminate their efforts as unfeasibl.i. In addition, no 

school system seems to have progressed past beginning efforts due to 

problems encountered in generating the quantity and quality of data 

21 

required and the need for explicitness of goals and objectives. 

The method of PPBS is, as was said earlier, extremely flexible. 

The central feature is the attitude it attempts to convey, which is the 
design and plans for programs and program alternatives based on consider- 
ation of the extent to which objectives will be met at various levels of 
funding, and, at the same time, the integration of the program activities 
over an extended time interval (e.g. 5 years). 

21 

Thomas Wilson and Shiv Gupta, ‘ Revtftw Of Research and Projects with 
Content Relevant to the»,§ti>dy. Unpublished dooument for the T^ter-aediate 
Unit Planning Study, of Local and State Government, 

University of Pennsylvania, (Philadelphia: November, 1967), pp. 2-3. 
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One of the essential elements for a system of planning auch as 
this, in addition to a specification of goals, programs, and program 
objectives, is an accounting system which can relate costs to program 
activities. This represents a departure from the line-item accounting 
systems prevalent in school systems. Accrual accounting procedures are 
also important so that expenditures can be tied to time in a more realis- 
tic fashion. 

Some of the recommendations for PPBS indicate that systematic 
analysis, such as benefit-cost analysis or cost-effectiveness analysis, 
should be included within the scope of PPBS. The elements attributed 
to PPBS are the same as those which generally define (for example) 
benefit-cost analysis — objectives, alternatives and selection from 
these alternatives. 

In summary, it appears that PPBS is no more than the label assigned 
to efforts aimed at applying systematic analysis (e.g. cost-effectiveness 
ana lysis) to ongoing systems in a fixed time frame. 

4. Input-Output Analysis 

This work derives from the input-output studies of Leontief . As 

was mentioned earlier, the efforts to relate inputs to outputs in 

education are empirical in nature, and almost totally unsupported by 

theory. Some studies have attempted to relate educational inputs (e.g. 

achievement test scores; delinquency rates) in an effort to determine 

22 

the technical coefficients which link them. 

22por example: The Feasibility of Cost-Effectiveness Analysis for 
Title I Public Law 8 9 - io . tecnnomlcs , Inc . (Santa Monica: lyoo). 

Thomas, James, J. Alan Thomas and Harold J. Dyck, Wealth Expen- 
diture and Decision-Making for Education . School of Education, Stanford 
University (Stanford: June 30, 1963). 
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While the Kershaw and McKean article seems to be the first of the 

proposals from non-educators to study input-output relationships, the 

work of Paul Mort predates this. Mort examined levels of expenditures 

and various policy positions in relation to criteria he designed 

reflecting school system "quality." Mort was able to describe gross 

24 

relationships and draw conclusions. 

Kershaw and McKean wanted to examine input combinations and their 

effects (or, more accurately, associations) with school outputs. They 

proposed that empirical input-output studies be conducted, and, if the 

findings produced suitable hypotheses, that these hypotheses be tested 

by means of alternative educational systems. 

Roger Sisson at the University of Pennsylvania has approached this 

problem, but he has paused to place more emphasis on the process by 

which inputs help produce outputs. His work in this area also considers 

25 

models of the school system. 



9 3 

Joseph A. Kershaw and Roland N. McKean, Systems Analysis and 
Education , (Santa Monica: The Rand Corporation, 1959). 

^^In 1946, Mort, William S. Vincent, and Clarence A. Newell develop- 
ed "The Growing Edge',' which was an instrument designed to measure the 
quality of education in specific school environments. Many studies and 
dissertations were published by the Bureau of Publications of Columbia 
University as a result of the efforts of Mort and his colleagues. They 
were primarily empirical studies concerning themselves with the 
sensitivity of school quality to alternative input levels, 

^^Roger L. Sisson, A Model of a School , Educational Intermediate 
Unit Study, Management Science Center, University of Pennsylvania, 
(Philadelphia: September, 1968); Roger!. Sisson, "Can We Model The 
Educational Process?" paper presented to Sjmiposium on Operations 
Analysis of Education for U. S. Office of Education, (Washington, D* C.: 
December 1967) . 
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5. Summary Comments on Selected Methods 
Benefit-cost analysis seems appropriate for selecting projects from 
among alternatives when investment is clearly the spirit of the decision, 



and when 



xiipUtS 



eind 



utputs can fairly be measured in dollar terms. Cost- 



effectiveness analysis appears to be applicable to the selection of 
alternative designs aimed ''t the same goal or goals. In addition, an 
effectiveness measure substitutes for a benefit measure in the analysis, 
and, generally, time is treated as less important. 

Program Planning Budget Systems are attempts to bring systematic 
analysis to ongoing systems by integrating programs and program plans 
with consideration of the gains and costs of the activities. This is 
reflected by a 5-year budget, which is revised annually. 

Input-output analysis is openly aimed at gaining knowledge by 
studying which input combinations are associated with the presence of 
various desired outputs. The endeavor could probably gain significantly 
if more attention would be extended to the relations that exist between 
inputs and process on one hand, and process and outputs on the other 
hand. Input-to-process and proc*e-to- output .relationships are fundamental 
to explaining why certain inputs are related to certain outputs. The 
process, it can be hypothesized, may, in some instances, be the major 
explanatory factor in the analysis. 

Exhibit 1 describes these previously discussed methodological 

approaches which have been suggested for application to education 

26 

problems. The message that filters out of the literature is that what 



26 

^Research for Better Schools, Inc. , published in November ,1968 , a 
bibliography titled, An Annotated Bibliography of Benefits and Costs in 

the Public Sector . This publication includes approximately 2700 referen- 
ces and comments on articles , books, reports, etc. , in the domain of 
economic and quantitative analysis in the public sector. 
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is needed for education is a systematic method for evaluation of ongoing 
educational systems so that future-period allocations can be made with 
full awareness of the appropriate decision inputs. 

Chapter II structures and analyzes nine decision frameworks. The 
analyj3is leads up to Case 9, which is the evaluation of ongoing systems 
pursuing multiple objectives by means of multiple activities. 

Exhibit 1 



Comparison of Selected Public Sector Analytic Methods 



General Problem Benefit 

Method of Analysis Framework Measure 



Planning 

Horizon 



Benefit-Cost 



A priori selection 
of projects 



Discounted An arbilrary 
net dollar horizon based on 
benefits project factors 



Cost-Effectiveness A priori selection 

of systems 



Effective 

ness 



A complete 
operation of 
the system 



PPBS 



A priori with annual 
feedback for up- 
dating programs and 
budget packages 



A variety Generally five 
of measures years 
e .g. Benefit- 
Cost 

Cost-Utility 



Input-Output 



A posteriori 



Usually multiple Variable 
output criteria from one year 

to several 
years, de- 
pending on 
purposes . 



II. RELATED DECISION CASES 



This chapter describes a set of related decision situations. Nine 
cases are developed and criteria are specified to assist selection 
decisions in the a priori cases, and to improve allocations for existing 
programs in the a^ posteriori cases. The purpose of the analysis is to 
structure a general model framework which is applicable to the evaluation 
of ongoing systems. The next chapter treats the specific problems of 
adapting and applying ^he general evaluation model to an ongoing educa- 
tional system. 

Exhibit 2 outlines the logic of the classification system. It is 
interesting to note that within the system of classification only one of 
the nine cases (Case 8) is a cost-effectiveness decision case and one 
(Case 9) is a cost-effectiveness case, in an after-the-fact evaluation 
sense. 

A. The Nine Cases 

The format for the development and analysis of each case is given 
below: 

1. Case structure 

a) The number of objectives (single or multiple) 

b) The number of activities (single or multiple) 

c) The decision framework (a priori , i.e. activities are to be 
selected or £ posteriori , i.e. activities were selected and 
are to be evaluated) 

d) The level of resources (unlimited, limited, or known) 

e) A discussion and a statement of the decision-maker's 
objective 
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EXHIBIT 2 



A CASE CLASSIFICATION OF LOGICALLY RELATED DECISION PROBLEMS 



STRUCTURE 



CASE 



DECISION 

FRAMEWORK 



RESOURCE 

LEVEL 



COMMENT 



1. Single objective with A priori 
set of activity-designs; 

one to be selected. 

2. Single objective with A priori 
set of activity-designs; 

one to be selected. 



Unlimited A utopian research and 

development problem. 

Limited The constraint limits 

admissable alternatives; 
but still no incentive 
to economize. 



3. Evaluation of Cases 
1 and 2. 



A posteriori Known 



A performance evalua- 
tion involving a par- 
tition of outcome 
space. 



4. Single objective with A priori 
set of activity segments; 
several to be selected 
as a package. 



Unlimited A more complex version 
of Case l;still 
utopian. 



5. Single objective with 
set of activity seg- 
ments; several to be 
selected as a package 

6. Evaluation of Cases 
4 and 5. 

7. Multiple objectives 
with sets of activ- 
ities; several to be 
selected. 



8. Multiple objectives 
with sets of activ- 
ities; several to be 
selected. 



9. Evaluation of Case 8 



A priori Limited 



A posteriori Known 
A priori Unlimited 



A priori Limited 



A posteriori Known 



The constraint limits 
admissable alternatives 
as in Case 2;still no 
incentive to economize. 

A more complex version 
of Case 3. 

A much more complex 
version of Cases 1 & 4; 
the relative weight of 
objectives becomes 
important. 

The general cost-effec- 
tiveness case; the only 
case meeting the neces- 
sary and sufficient con- 
ditions for cost-effec- 
tiveness analysis. 

The general program 
evalur?tion case; pro- 
vides cost-effective- 
ness evaluations for 
present year and inputs 
for next year's budget. 
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2. Case analysis 

The decision- variable is structured and analyzed. A proof is 



given, when necessary, to show that the preferred value of the 
decision-variable is best in terms of meeting the objective(s) of 
the decision-maker, given the case assumptions and conditions. 

Case 1 

This situation is described by a single objective for which the 
decision-maker is presented a set of alternative activity-designs pre- 
pared by his engineers. An activity -design is a proposed plan for an 
activity. The decision framework is £ priori and resources are unlimited. 

Only one activity-design can be se'^cted by the decision-maker. 

27 

This does not constitute a cosu-ef fectiveness problem in that 
resources are unlimited. Instead, the decision-maker is confronted 
with a utopian research and development problem for which a solution 
is to be engineered. 

The apparent criterion is to select that alternative promising the 
highest expected performance, Ki. Activity-design i has a distribution 
of performance estimates, K^, which reflect the engineer s perception 
of the performance after implementation and the engineer s biases. A 
bias for or against a particular activity-design affects but not the 
standard deviation of the distribution of K^, since a constant added to 
or subtracted from every value of a variable cannot influence the 
standard deviation of the variable. The decision-maker’s aim is to 

^^The presumption is that benefits deriving from the pursuit of the 
objective may not be reducible to dollar terms, especially if intangibles 
are involved. Therefore, cost-effectiveness is general since it is 
independent of any arbitrary unit system. 
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select that alternative which promises a preferred balance betwee’i 
estimated average performance and the variability of the distribution. 

The standard deviation,^, has been used as a measure of uncertainty 
in some studies dealing with investment decision models. The key ideas 
center on the assumption that an increase in risk (variability) in the 
return on an investment can be made acceptable to the decision-maker, 
provided there is sufficient compensation in terms of an increase in the 
expected level of return,yl4. Also assumed is the notion that risk or 
uncertainty is undesirable. A certainty-equivalent function Is an 



which the decision-maker is indifferent. Specificity is given to an 
indifference curve by the certainty equivalent (usually a monetary value). 



on a curve or elect to receive the monetary value for the curve, he 
should be indifferent to all. 

Two comments are in order. First, risk and uncertainty were used 
synonymously in this description. Second, the usual way indifference 
curves are used is to indicate the rate of exchange for two commodities 
as a function of a set of preferences. The curve is negatively sloped 
reflecting that it takes an increased amount of one commodity to offset 
the loss Oi. an amount of the other commodity. The slope of the curve 
at any point is called the marginal rate of substitution. 

28 

For example, Donald E. Farrar, The Investment Decision Under Un - 
certainty , (Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1962), 
pp. 25-26. 



indifference curve that is the locus of combinations 




. for 



If the decision-maker is allowed to choose from among 
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How does this relate to the selection of the activity-design? 
Assume the engineer can provide, for each proposed activity-design i, 
an estimate of and where 



It is assumed that performance is defined over the range 0 to 1.0. 



is based on the engineer* s estimate of a level of performance which 
"cannot" be surpassed by the activity-design. Once Kq has been estab- 
lished, the decision-maker considers variability values which are equally 
preferred to the receipt of Kq. These points define an indifference 
curve which is independent of any data received from the engineer. 

That is, the curve is defined solely on the basis of the decision-maker's 

29 

preference for combinations of the two variables. 

Figure 1 indicates the general nature of this indifference curve. 



performance values to variability values. This is explained by the 
assumption that the decision-maker would elect to avoid rist , Therefore, 
as S increases the response required of K is greater in order for the 
decision-maker to remain indifferent. In the extreme, there is a value 

29if the preferences of the decision-maker were not used then he 
would be unnecessary and the engineer could make the selection strictly 
on his own criteria. 



and 




The initial task for the decision-maker is to select a level of perfor- 
mance, Kq, below which performance is unacceptable. The upper limit 



denoted I^^ y The curve indicates the relationship of average 
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of S as perceived by the decision-maker which is so high that he will 
not accept anything less than the receipt of perfect performance with 
certainty. It should also be pointed out that two indifference curves 
cannot intersect, since this would mean there are two certainty equiva- 
lents for the same point. 




Variability (S) 

Figure 1. Indifference Curve of 



The X region of Figure 1 contains points for activity-designs 

which have expected performance levels below the minimum acceptable 
30 

level, Kq. The Y region contains points that are acceptable with 
respect to expected performance, but due to the assumption that increases 
in variability require increasingly greater responses in performance, 
these points are dominated by any point on I^j^ 

Figure 2 shows a group of indifference curves. The curve I(Ko) 
indicates a sharp response to what presumably is the decision-mrker' s 
uneasiness over the high level of variability in the neighborhood of 
S = 0.5. Suppose a parabola of form S = a + bK + cK^ is used to describe 



30 



They also lie on indifference curves with certainty equivalents 
less than Kq, since the curves cannot intersect. 
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these relationships. Since three constants are to be determined, three sets 



Performance (Kj^) 




Figure 2. Several Indifference Curves 

of coordinates are required. For instance, ^Kq = 0.4, Si= 0^ , 

= 0.6, Si = 0.3^, and = 1.00, Sj * 0.6^ satisfy the function 

Si = 0.10-0.75K^ + 1.25 k 2. In general, Ki values can be obtained for 

the quadratic form using the formula 

= -b + yb2 - 4c(a-Sj^) ^ ^ ^ 0 , 

2c 



Some checks on the equation would involve seeing that 1 

and Ki$l. The slope of the function can be interpreted as the 

- ^ 

rate of substitution of for S^, that is, ^jg ; 

K * -b + /b^ - 4c(a-S) * Tc i ^ -4c(a-S)]|^ 



2c 



M =(— L.)(—L.) -4c(a-S) 3”'^(4c) 

dS 2c 2 *• 

— * + fb^ -4c(a“S) 3 
dS “ ^ 

This indicates that the change in performance per unit of varia- 
bility at the point S = 0.5 in the previous function is 0.53 (-0.53 is 
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meaningful). This means that the rate of change of the function when 
S = 0.5® is 0.53 K per unit of S. At S = 0.6 the derivative is 0.57 
indicating the increase in the response of K to changes in S. 



Performance (Kj^) 




Figure 3. Selecting a Preferred Point. 



The logic that allovjs the preferred point to be selected is not 
complex. The ideal point is = IjS.^^O^ . Figure 3 shows an indif- 
ference curve and three points. Points A and B are equally preferred 
since they lie on I^j^^i^Point C is preferred to A or B since it has a 
higher performance value than some points on I(k’) having the same 
level of variability as Point C. Point C is also preferred to any 
point on having the same level of performance as C. The preferred 

activity-design is the one with a point lying on or above the uppermost 
indifference curve. 

The main lesson of Case 1 is that the decision-maker should intro- 
duce his preferences as to combinations of performance and variability. 
If not, expected performance is the criterion and his preferences for 
risk are ignored. 
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Case 2 

This case differs from Case 1 in one important respect: resources 
are limited. Again there is a single objective with multiple activity- 
designs from which only one is to be selected in an a priori framework. 
If the decision-maker is being perfectly rational, he will select that 
activity-design which promises the highest certainty equivalent he 
can afford. When he does not elect to behave in this manner then he has 
implicitly introduced a second objective. It may be a personal objective 
such as the desire to be efficient under any circumstances or he may 
entertain some vague notion of future possibilities developing for 
investment. But the point remains that in Case 2 he has no alternative 
goals in competition with the attainment of the single, given objective. 

Consequently, Case 2 reverts to a modified Case 1. His decision 
rule is to select the activity-design represented by the point on or 
above the highest indifference curve provided it falls within the limita- 
tions of his b»:dget. Instead of treating the three variables in a three 
dimensional drawing, the decision graph can be viewed as two dimensional 
by using the certainty equivalent in place of and S^. Figure 4 shows 
this with the ordinate indexing values of the certainty equivalents and 
the abscissa indicating estimated costs associated with the candidate 



^^Costs are assumed to be point estimates with no accompanying mea- 
sure of variability. Therefore, the decision-maker should be cognizant of 
this since designs with estimates less than, but close to C^ax exceed 
Cmax they are selected. How he views this is a function of how 
binding Cmax is to him. If, for instance, there is some room for leeway 
he may not be troubled by the possibility of exceeding Cmax* This is 
especially true in subsequent cases since provision for this contingency 
is generally in the overall budget structure. 
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activity-designs. The candidate points are points for which 

and In order to select the preferred point the decision-maker 




Figure 4. Selecting a Preferred Point with a Cost Constraint 

locates the point on or above the highest certainty equivalent with a 
C<Cxnax- If there are two or more candidate points on the highest line 
then the decision-maker can select the least cost point (although 
according to the structure and assumptions of Case 2 he is unable to 
use a partially unexpended budget). The net result of adding the budget 
limitation is that points such as B in Figure 4 are eliminated from 
consideration. In the Figure 4 example he would be compelled to select 
Point C over Point B if Point A did not exist. This indicates the 
potential importance of a budgetary restriction. 

Case 3 

Situations which are characterized by the structures of Cases 1 
and 2 result in the selection and implementation of an activity-design. 
Soon the decision-maker is faced with an evaluv’itive problem — given 
that the activity has been conducted, how well did it do? Case 3, there- 
fore, is the a^ posteriori evaluation of Cases 1 and 2. 
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At first it might appear that the problem should be structured as 
a test of a statistical hypothesis. This is not required, however, 
since possible performance outcomes can be partitioned without statistical 
consideration. Figure 5b shows the range of possible performance out- 
comes partitioned into several decision segments. In order to appreciate 
this partition it is necessary to consider the distribution of as 
perceived, £ priori ; by the engineer. This is shown in Figure 5a. 



could be attributable to factors which he could not possibly predict a 



priori . It should also be noted that Kq is a certainty equivalent and 

* 

the probability of K^< is dependent in part on the density of 



1.0 




0 




1.0 



Performance (Ki) 



Figure 5a. Prior Distribution of 



0 




Performance Outcome (K|) 






*ic 

Figure 5b. Posterior Partition of 



When actual performance, Kj, falls below the level of (i.e. 

Vlf 

OXK^^Kq), the engineer has not necessarily made a miscalculation. This 
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The decision-maker may re-evaluate the engineer's ability to design and 
estimate and the engineer may evaluate his methods, but the previous 
cautions should be noted. 

If actual performance, K^, falls between and^^Cthe engineer 
considered this to be certain before the activity was implemented), then 
it is within the range deemed acceptable. The engineer should evaluate 
his methods to be sure that he has minimized the probability of getting 
the right answer for the wrong reason. 

When actual performance exceeds the upper boundary of the prior 
distribution (i.e.^^^ K^^l.O), then the engineer has miscalculated 
on the favorable side. Nevertheless, this "pessimism" could rule out 
other activity-designs which should be selected for implementation. The 
decision-maker should re-evaluate the engineer's ability to estimate 
and the engineer should evaluate his methods. 

When considering actual outcomes, the decision-maker should under- 
stand that the main benefits derive from discovering improved methods 
for 1. designing future activities, 

2. estiroat ing the potential for what is designed in the future, and 

3. implementing future activities as they are designed. 

The problems of implementing a newly designed activity often require a 
technology that is not understood. A preferred system can be perfect in 
terms of technical design, yet the implementation people may not be able 

32 

to introduce the initial parts due to the nature of the changes required. 



A few educational organizations are studying the concept of change 
with an objective of developing a technology for school systems that 
enablas them to bring about the changes they deem necessary in a more 
orderly fashion. 



- 31 - 



The reason for introducing this apparent digression on change is that 
it is fully possible that the engineer can design an activity with good 
potential only to see performance fall below due to problems arising 
out of the way the operation was implemented . Better communication 
between the engineer and the implementer may provide some help. 



Case 4 

Cases 4, 5, and 6 closely parallel Cases 1, 2, and 3. The differ- 
ence is that Cases 4, 5, and 6 permit the decision-maker to select a 
package of activity segments in order to achieve his single objective. 

Case 4, then, considers a decision-maker pursuing a single objective 
and having unlimited resources. The framework is a priori and he is 
able to select a set of activity segments. 

Previously, there was an implied assumption that, in the interval 
Kjj to X^, changes in worth to the decision-maker were proportional to 
changes in nerf ormance That is, 

A-^- " -*£bC_ , where, 

A «AB «'BC A«XY 

A^ab * the change in performance from to Kg , 

where both points are between and and 

** change in worth associated with^K^g. 



^^It can also be assumed that below Ko(the minimum level of per- 
formance set by the decision-maker), the same performance -worth relation- 
ship exists, but the decision-maker feels that an inadequate level of 
worth would result for performances in this interval. The same reasoning 
holds for K!>»1 as in the interval . This assumption is carried through 
Case 6. 
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The reason for introducing the notion of worth, is that the 

respective segments within an activity-design do not contribute equally 
to the overall objective. Worth will be used to weight performance, and 
expected weighted performance will serve as the decision-variable. 

The situation confronting the decision-maker of Case 4 is that he 
requires a package of activity segments . Some of the segment outputs 
are independent of the outputs of previous segments and others are not. 

The decision-maker can ask his engineers to repackage segments in any 
way possible in order to generate an improved way of attacking the 
objective. In addition, the decision-maker perceives segments as 
potentially contributing unequally to the overall objective. The 
problem is to select that package of segments promising the largest 
expected worth . 

Theoretically, the engineer would structure a multivariate joint 
density function describing the outcomes and associated probabilities 
for each activity-design or package, 

^^^il»^i2» • " • »^lm^ 

In the statement above would refer to the minimum performance 

acceptable for certain for segment J of activity-design i, and ^ j,j 
would refer to the "maximum performance possible" for segment J of 
activity-design i. 

Then the engineer would be able to derive marginal density functions 
by integrating the joint density with respect to the other variables over 
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their full ranges (e.g. to ^ij for tho general term)* For instance, 



the marginal density of segment 1 is 
V 2 Xi3 \m 

fl(Kii) f ••• f f • »^im)^%2^^i3* * ’‘^^im» 



/ / 



^2^'^i 3 ^im 



fj(Kij)> 0. 



Also the joint distribution of segments 1 and 2 is 

»13 "Xi4 im 



gi(Kii.Ki2) = y f'J 



^0i3 ^oi4 ^oim 



,K£2» • • • > ^im^*^^f3^^i4* * **^^im» 

fj(Kij)iO. 



If the outcomes of segment 2 are independent of segment 1 outcomes, 
then their joint density function vould be equal to the product of their 
marginal densities. From gl(Kil>Ki2) engineer would obtain the 
conditional density of segment 2 outcomes given those of segment 1 by 
regarding as a constant over its range of values, This is 
defined as 

. 8l(Kil,Ki2) . 

hoCKia! ^ii) » fi (Kii)^ 0. 

fi(^^ii ) 

As was indicated in footnote 35 the engineer could, if the decis' on- 
maker requested, answer such questions as ”what is the probability that 
performance in segment 2 will take on a value K^2 more given that 



o / 

^In a general sense, independence can be assumed when the joint 
density can be factored into marginal densities each involving only one 
variable and with the limits of each not involving another of the vari- 
ables . 

35 

Or for that matter, over any part of its range. If the range of 
Kii is restricted then the conditional density of will be restricted 
to being conditioned upon outcomes of in the tes trie ted range only. 
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performance in segment 1 was also or more?" This would be structured 



variate density level down to the univariate level if the engineer were 
capable of providing a distribution of such complexity. In all fairness 
to the engineer, however, it is assumed for the remainder of this 
analysis that he cannot develop distributions more complex than bivariate 
densities. 

Since the decision-maker wants to be able to compare alternative 
packages of segments so as to select that package promising the highest 
expected worth, he must consider several things. He has the engineer 
derive the marginal probability density for each segment from the bi- 
variate density linking each segment and the preceding segment. Then 



'' The implicit assumption is that dependency is an appropriate con- 
sideration .^or adjacent segments only. This means, for instance, if 
segment 3 outcome^ depend on those of segment 1 the dependency would be 
ignored by this analysis. A more general case would involve considering 
all segment combinations. In this instance^ C™ relationships would re- 
sult but as will be seen later, expected worth is a function of the 
marginal distribution alone. Consequently, the extension is only appro- 
priate when considering particular probability questions. 




8i(Kii,Ki2)dKii ,dKi2 





ii 



fj(Kij)dK-i 



And if the sequential outcomes are independent, then 




The preceding type of analysis would be possible from the multi 
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he assigns a worth value Wj^j to each segment, where 

Lastly, he must consider the relationship of w^j to performance outcomes', 
K^j , of the respective segments. 

The procedure involves taking the respective bivariate densities 
and integrating out the marginal probability density, f j(K£j) , for each 
segment. Then fjCK^j) is transformed into Wj(w^j), and the expected 
worth for the entire package can be found by aggregating over the respec- 
tive segments in the package. For the sake of illustration, let the 
following information be given for activity-design i. 

8l(Kil,Ki2) 

82(Ki2»Ki3) 

When these bivariate densities are factored into their component 
probability densities 



8l(Kil,Ki2) 
82(^12 >^£3) 



= fl(Kii) f2(Ki2) 

- f2(Ki2) hsCKisI Ki2), 

= f3(Ki3) h2(Ki2|Ki3). 



The 



transformation of into is linear in all cases and is 



and 



Wil = biiKii + aji 

®12 “ *’i2*^i2 + Hi . 

Wi3 = bj^3Ki3 + a^3 

Solving for K^j in terms of W£j, we obtain 

Kil = ^^il-ajl 
bil 

Ki2 - Wi2“aj2 
bi2 

^i3 * ’"i3"^i3 » 

^i3 



0£wi2i%2 

0i.Wi3 

bil>0, 

b£2 ^0, 



and 



The differentials of j are 

KT 

dK^2 " dw£2 ’ 

bi2 

dKi3 = dw£3 . 

^i3 



‘o 

ERIC 

P— 



Since the outcomes of segment 2 are assumed to be independent of 

segment 1 outcomes, the expected worth integration involves obtaining 

the marginal densities, transforming performance into worth, and 

generating the first moment. The marginal density of is 

and it is a probability distribution. That is 
‘il 




fl(Kii)dKi = 1, 



f^(Kii)20 



The marginal density of Wj^^ is , and it is a probability 

37 

distribution. That is 

J l (w^ j^)dw^2^ = 1, 0. 

_ il^Oix+^ii 

The expected worth, of segment 1 is 

^il ^il+^il 



w.^= J WiiWi(wi 



‘?il)dWil • 






^^The limits of integration for W^, are not, in general, 0 and 
since the scale of Kii ranges from 0 to 1.0 With Ko^^and generally, 

viithin this interval. 
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The identical procedure for segment 2 will yield w^2» segment 

3, which is related to segment 2 in terms of performance outcomes , requires 
a consideration of the conditional relationship. The expecteu value of 
KfS given a particular value of K^2 arithmetic mean. If, however, 



this is a function of K^2 ^nd what is needed is the expectation of the 
distribution of expectations. 

That is, 

I)E(yIx) is in general a function of x. Let u(x) denote this 
function and h(x) be the probability density of x. Also X and Y 
denote the random variables and x and y values of the random 
variables . 



expectation of the raargii.al density of Y. Consequently, w ^3 is equal 
to the expected value of ‘ From this example it is seen that. 



^^Alexander M. Mood and Franklin A. Graybill, Introduction to the 
Theory of Statistics , 2nd edit. (New York: McGraw Hill Book Company, 
Inc.j 1963), p. 158. 



the expectation of Ki 3 over the entire range of K^2 desired, then 



2) e[e(y|x) 3 * E [u(x)J 




■ir g(y|x)h(x)dydx 
= E(Y) = y 



y g(y|x)h(x)dydx 



OQ 

The proof is taken from Mood and Graybill and it shows that the 
conditional expectation of y)x is equal, in general, to the mathematical 
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in general, the expected worth of activity-design i is the sum of the 
expected worths for the segments in the activity-design. That is. 



As in Case 1, the decision-maker compares alternative packages and 



this instance it is the overall expected worth since he is unconstrained 
by resources and does not have alternative outlets for expenditures. 

Case 4 has introduced several features into the analysis. The certainty 
equivalence str’^.cture was abandoned due to the advantages of using 
probability distributions for multivariate situations. In addition, 
mathematical expectation played a critical role since the need to reflect 
the unequal potential contribution of segments was apparent. The 
decision variable, weighted performance, was obtained by means of 
transformation. While the transformation was treated as linear, the 
analysis is not restricted to linear transformations. The perspective 
needed to transform performance into worth, in a given situation, can 
only be supplied through an understanding of the real importance of 
performance outcomes to the desired outputs of the enterprise. 

“50 ■— 

Implied in the summation of w^. is an assumption that the scale 
of Wj^j is an interval scale. This means the scale is unique up to a 
positive linear transformation. That is, a new scale, Y, can be made 
by a linear transformation of the X scale (y * a + bx; b»0) and not 
distort the underlying relationships. See, for instance, Torgerson, 
Warren, Theory and Methods of Scaling , (New York: John Wiley & Sons, Inc. , 
1958), pp. 19-20. 
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selects the one with the preferred value of the decision variable. In 
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Case 5 

4 

This case brings a cost restriction to the structure of Case 4. 

Here the decision-maker strives for a single objective by selecting one 
activity-design (i.e. package of segments) in an £ priori framework. 

Again, as in Case 2, the problem does not have cost-effectiveness 
overtones since there are no alternative ways to use unexpended resources. 
The only restriction theJ budget constraint places on the decision-maker 
is that the package he selects on the basis of the expected worth 
criterion may have a lesser mathematical expectation than that of Case 4. 
This argument is completely consistent with the Case 1-2 parallel. 

Graphically, this is seen in Figure 6 which is structured identically 
to Figure 4. The difference between the two figures is that in Figure 6 
the ordinate indexes expected worth, while Figure 4*s ordinate indexes 
values of the certainty equivalent. The reasoning is perfectly 
analogous. The candidate points lie in the region where C^ 5 Cmax 
^ij^^(Ko) , Before continuing, it is appropriate to define 

^(Ko) • ^ segment had a minimum perfor- 

mance level which was set by the decision-maker and the engineer. This 



value, Kq^j, was actually a concept carried forward from the certainty 



Total 

Expected _ 

Perf ormance ) 




Figure 6. Selecting a Preferred Package with a Cost Constraint 
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equivalent analysis of earlier cases. At a higher order level there 
is still an overall which the decision-maker sets. It was assumed 
that Wq£j “ was the transformed value of Therefore W q^j 

is assumed to be equivalent to the worth corresponding to the overall 

tn 

Yi . This implies that Jwq. is a constant for all activity-designs, 

J=1 

although the condition is not necessary for the analysis. 

An additional complication arises in that the decision-maker 
considers the advisability of allocating the budget among the segments 
of the preferred accivity design. (The stance taken here is that this 
consideration shoi.ld be delegated to the manager of the implementation 
team. The feeling is that this is fundamental and in the best interests 
of all concerned, since it is the implementation manager who accepts the 
implementation responsibilities.) The problems associated with the 
human interfaces between the decision-maker, the engineer, and the 
implementation manager hrve been referred to earlier and are brought out 
again. 

Case 6 

Situations which are characterized by the structures of Cases 4 
and 5 result in the delection and implementation of an activity-design. 
At the end of a period of time the decision-maker is faced with an 
evaluation problem -^-given that the activity-design has been put into 
operation, how well did it do? Case 6, therefore, is the a posteriori 
evaluation of Cases 4 and 5. 

The rationale employed in this evaluation is similar to that of 
Case 3. The purposes of the evaluation are to discover improved ways of 
designing activities, estimating the potential performance of the 
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design and implementing the package as it was designed. 

The present case has two levels of evaluation. At the higher 



order or overall level the decision-maker wants to look at the overall 
performance and compare it to K^. It is allowable to think in terms or 
performance or worth depending on the type of question being considered. 
If an overall com.parison is required, it is probably more satisfying to 
transform the produced worth, based on the observed performance outcomes 
of the respective segments, into an overall performance variable, K* • 
Two points of this relationship are given from the analysis. They are 
overall performance produced worth 



maker in order for him to be able to make the overall evaluation. 

For the most part this evaluation is an overall comment on the manager 
of the implementation team. The engineer also is interested in the 
outcomes, since it was his design that was implemented. The interrela- 
tionship of responsibilities can cause problems for all parties involved. 
Therefore, the emphasis should ba on Improvement of methods rather than 
assignment of blame. 

The second level of evaluation focuses on performance of the 




\ 




The points connecting 




w* and K* are required of the decision- 



respective activity segments. Here the emphasis is directed more to 
the engineer’s estimating methods, the decision-maker’s worth assignments, 
and the members of the implementation team. These evaluations are 
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j 1. 40 

similar to those of Case 3 and are not repeated here. 

Case 7 

The structure of Case 7 introduces multiple objectives which will 
be referred to as overall system objectives. The decision-maker if faced 
with the a prior i selection of activity-designs with unconstrained 
resources . 

For the first time, there are alternative competing objectives but 
since there is no restriction on level of resources, the decision-maker 
will select a system of activity-designs which, when implemented, 
should contribute as much as possible to the overall system objectives. 

As was implied in Cases 1 and 4, the limitations on ultimate system 
performance are a function of the engineer’s imagination, the state of 
the technological arts and the abilities of the implementation team. 

Let 0^, Og,..., Ck ...,0g denote the overall system objectives. 

Since the activities selected will ultimately be evaluated in the basis 
of how well they contribute to these objectives, it is important to 



^®Each segment has its own probability density. Since only one 
outcome is observed, the distribution of outcomes for segment J is 
f 7 (K. J. This, however, is from an a priori point of view. The fact 
tiiat some ’’impossible" outcomes can occur, i.e. K?j >kj " , 

is a clear indication that the notion of a posterior distribution 
of K*j is appropriate. Bayesian extensions to the analysis of the 
nine cases are not developed, but further work within this type of 
framework could include such analysis. 
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value them^^. The values afford the decision-maker the opportunity to 
consider trade-offs for the overall system objectives. The common 
frame of reference or dimension is, of course, value. The assumptions 
underlying this assignment of values are identical to those for assignment 
of worth in Case 4. This value function supplants the worth functioi 
which was used to weight outcomes so that a common frame of reference 
could be developed. The value function, developed in Case 7, provides 
a uniform basis for assessing outputs prior to, as well as subsequent 
to, the operation of these activities. These values are denoted V(Oa), 

V(0b),...,V(0j^), V(Or). 

Each activity or activity segment has a potential for contributing 
to one or more of the overall system objectives. This indicates that 
the value of overall objective 0. is the maximum that can be produced by 
the system with respect to the end. If all overall objectives were 
produced perfectly, then the system would have produced the maximum value 



The extent to which a particular activity contributes to a particular 
overall system objective is a function of two variables -- potential for 



^^Russell Ackoff ,et.al. , show that for systems with two mutually ex- 
clusive, totally exhaustive objectives , the maximization of expected value 
is accomplished by maximizing performance. (Actually they label perfor- 
mance as efficiency and define expected value asT Vj, whereEij is 
performance on objective J by method i and Vj is -J the value of objective 
J) «An extensive proof is given for the case with three or more objectives. 
This maximization requires performance and values. 

Russell L. Ackoff, Shiv K. Gupta, and J. Sayer Minas, Scientific Method , 
(New York: John Wiley & Sons, Inc., 1962), pp. 93-97. 



possible , i . e. , 




contribution and actual performance. That is, Gm/L 
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where 




= the effectiveness of the mth activity toward the 
Lth overall system objective. 



and 



V(ajjj^ the maximum value that the mth activity could produce 
with respect to the i-.^h overall system objective. 




= the adjusted performance score. 



The maximum value, V(a^^) , that the m*-^ activity could produce with 
respect to the overall system objective is a function of how impor- 
tant that activity is relative to the other activities aiming at (>^. 

Here the decision-maker makes a relative value assignment such that the 
sum of the relatives is equal to V(Q;). s'ere there only one overall 
objective, there would be no need for V(an^^ and Case 7 would reduce 
to Case 4. The distribution of overall value to the activities is 
represented as follows. 

AcMvitv aittipd at Oi Relative Maximum 

AC-ivicy aimea at importance value possible 

/(aii;v(0^) = V(aU) 

= ''(a,.) 




• • • • 

Based on this distribution of potential value, V(0* ) , to the activ- 



ities, it is possible to determine effectiveness at the activity level. 
In order to do this several performance concepts need definition. 

The decision-maker and the engineer establish a criterion that can 
be used to assess the activity’s performance. Then they list the pos- 
sible outcomes and relate these possibilities to index scores for the 
criterion. For instance, let us suppose that the activity was aimed 
at "getting eligible people to pass Examination Five". One criterion 
might be the number of eligible people passing the examination expressed 
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as a percentage of the total number of eligible people. The index could 
be formed by the percentage points. In this context, is the index 
score (percentage) reflecting the performance of activity m toward over- 
all system objective iu The decision-maker, however, may not feel that 
each percentage point is entitled to one percent of the potential value 

of a ;. In this case K*. is transformed into K^. The relationship 
mh mA. 

between and depicted in Figure 7. When K*^ is 

brought together with V(a^j^) the result is E* ^ or effectiveness. 
Effectiveness is defined as weighted performance. That is, * 



Transformed 
Per 

1.0 



V 

Performance (K_* ) 

lUp0. 



1.0 




Figure 7. Transformation of Performance Index Scores into Effectiveness 
Scores 



The right hand section of Figure 7 shows the relationship between 
the criterion index scores and the dec is ion-maker *s adjustments, 

K* ♦ The special case is where 1L; * iCxpver the complete range from 0 
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to 1.0. In general K the function k denotes the 

perceptions that underlie the transformation. 

The left hand section of Figure 7 shows the way is used to 
generate value produced or effectiveness. The reflection line joins the 
points {e* . = 0 , R* . = ol and f E* . = V(a_ •) ,K*. • = 1 f , and specifies 

that the change in effectiveness with respect to a unit change in per- 
formance is constant. The relationship is 



E*- = K* • V(a 
mji^ niX» ' m■^ 



and the change in effectiveness for a small change in performance is 

= V(a^i) . 



A *. 

va.lL 



The differential of effectiveness, dE* is equal to d^* . V(a • ) 

miii 

The reflection line did not necessarily have to be drawn at 45° to the 
X-axis. If it had made a 30° angle with the X-axis then 
dE^A _ ^ 

“ ^^^i^'‘^/o.5774 » since the tangent of 30° is 0.5774 



dK 



* . 
ni-t 



and the equation describing the relationship between the variables would 



have been 






E^= k; ^.0.5774 V(a^) . 



noc' 



The 45° reflection line adequately describes the way the variables relate 
to each other in this analysis. 

Now that effectiveness has been defined as weighted performance, 
two simple ideas should be considered. 

1. Are large effectiveness scores always preferred? 

2. What is the maximum effectiveness score? 



er|c 
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In order to conclude that the decision-maker prefers any larger E 

score to any smaller E score, several couanents are necessary. First, 

the relative values assigned to the overall system objectives represent 

3 ggt of targets for the decision-maker. If a unit of value is eouaily 

preferred, no matter what objective is being produced, the decision-maker 

clearly prefers a larger E to a smaller E in any case. If a unit of 

value on one objective is preferred to a unit of value on another 

objective then the assignment of values requires revision. It is, of 

course, possible to constrain the production of an overall objective in 

such a way that some production is preferred over other production. 

For example, the decision-maker could specify that overall objective 

0^ must be produced at some minimum level- or better, i.e. 

* 0.62Cv(Oi;l. This type of production constraint 
m 

could have efficiency implications for the system. It would have the 
same effect on the system operations as a policy since a policy can 
inhibit functioning with respect to given criteria in selected instances. 

The search for an optimal solution for this type of problem depends 
on such factors as the nature of the production function, the costs of 
production, performance, and values. 

The maximum E score is the sum of the valuations assigned to the 

& 

overall system objectives, i.e., 1 V(CX^. This is readily seen by 

inspecting the E function; “ 

E* ; ss v(a , and observing that when is 

equal to 1.0 (the maximum value since ^1.0), 

Therefore, under conditions of perfect performance * 1*0) » 

V(0‘)» and over all objectives the maximum E is ^ V(O^). 
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In this situation, as in Case 4, it is within the scope of the 
argument to treat performance as a random variable. This procedure 
is now outlined. 

The engineer provides the probability density of as in earlier 

if 

instances. In an £ priori sense the distribution of f is 
identical to f .(K :) . 'h^o transformations are required. The first 
a Hows scores to substitute for scores. Since a simple 

multiplicative form has been used to relate E to its components, the 
second transformation is constant for all such relationships and the 

ic 

probability density of can be inferred from That is. 



ES, ; V(a *) ) 






Since V(a^^ is a constant for all^^, the expected value, E*^-, 
is given by 



A* 



i^(V(a^X) ) = 1 



tm 

A 



- V = I K*^. V(a^) f^(KSi(V(a^^ )dK^(V(a^) 

)=0 



r* .i 






E£X = V = V(a^; 







A final set of relationships can be commented on prior to consider- 
ing Case 8. It was seen that effectiveness is a function of performance 
and potential value. Performance, however, is in turn a function of cost 
and non-cost factors. In fact, the very foundation of cost-effectiveness 
is tied up in these relationships and it remains for Case 8 to put 
things in order. 
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This situation represents an extension of Case 7 under a budgeting 
constraint* Here the decision-maker pursues a set of overal.1 system 
objectives by means of selecting a group of activity-designs in an 

42 

a priori framework. This is the first and only cost-effectiveness case. 

To arrive at the conclusion that Case 8 represents the only cost- 
effectiveness case, the previous analyses should have provided clues as 
to the necessary and sufficient conditions for cost-effectiveness 
analysis. Case 1 was not a cost-effectiveness situation because costs 
were not involved. In Case 2 it was observed that even costs were not 
sufficient since the decision-maker was interested in a single objective 
and he had no other alternative outlets for funds that could be saved 
due to economic considerations. Consequently, it is not until Case 8 
that the analysis finds budgetary restrictions and multiple outlets 
competing for this budget. The necessary and sufficient conditions for 
cost-effectiveness ana lysis to be appropriate for a decision structure are: 

1. More than one alternative outlet for the available resources. 

2. Less available resources than the alternative outlets can use 
ideally. A budget constraint is not sufficient unless it forces 

^^This point was made earlier, but it warrants repetition at this 
time. Cost-effectiveness is considered by the author to be the general 
way to approach this problem. This is due to the general incommensur- • 
ability and intangibility of benefits. In addition, if program evalua- 
tion for the purpose of improving allocations in the next period is 
desired (this comment anticipates the purpose of Case 9) then the 
relationship of value produced now to costs now is generally of primary 
importance. This does not preclude the possibility of specific excep- 
tions necessitating some form of benefit-cost analysis. 
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the curtailment of at least one activity. If an organization has 
enough resources to do everything it wants to do in the near future, 
then cost-effectiveness analysis in the present is not needed to 
support allocation decisions for the near future. 

Case 8 borrows the structure and analysis of Case 7. To this it 

brings the implications of cost. The critical relationship in the an-- 

a lysis is between performance, cost, and non-cost factors. That is, 

where: 

= The performance index score 

Cjj^jl. = The estimated cost for activity m as a producer of 0 
(this estimate is not treated as a random variable). 

Gjjj), = The non-cost factors associated with the performance 
of activity m as a producer of While these could 
be viewed as a function of C^^, they are assumed to 
be given and adequate in the analysis. They include 
the human factors, such as disposition toward the 
tasks, level of managerial knowhow, level of skills, 
and the technologiowl factors. 

Assume there are overall system objectives valued as V(Oj^), V(Og),..., 
V(0r). To interject some realism into the analysis assume that the 
production of is specified by the decision-maker to be at least 
The decision-maker has his engineers develop cost-effectiveness curves 
for each proposed activity. Figure 8 shows a hypothetical curve prepared 
by the engineer. This function indicates more than C-i dollars must be 

Effectiveness (E •) 

VdK 



0 

Cost 

Figure 8. An A Priori Cost-Effectiveness Curve. 



^%or instance, a budget constraint of $1,000,000 would probably 
not restrict the vacation plans of a married couple. 
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spent before anything is produced* When C2 dollars are spent the 



This implies that the factors are restricting further productivity. 
Implicit in the construction is a series of relationships. They are 
the relationships between 



If the system had been in operation for a period, then evaluation 
could be conducted for the purpose of obtaining cost-effectiveness 
points. These points would reflect the productivity of the system ^ 
it is . Allocations for the next period could be based on these con- 
siderations. 



The problem involves non-linear programming with the elements of 
the situation falling into three classes: 1) the functions relating 
effectiveness to cost, e.g., specifications 

(constraints) telling how much of which objectives should be produced. 



ming problem would involve a numerical analysis of the functions. This 
could be done by taking the derivative of effectiveness with respect to 
cost at, for example, $100 intervals for each proposed activity. This 
would generate a list of slopes which can be combined for all activities. 
Then the production inequalities (those specifying production must be at 
least »a) can be satisfied. Once this is done the remainder of the 
problem is a search for the maximum production for the unencumbered 
funds. This is not as simple as the procedure just described because 
the decision-maker cannot purchase the increment of effectiveness 



marginal productivity response becomes almost zero at the level ofE;, 



and 




^ * 

e.g. and 3) the budget constraint, i.e., m program 



ERIC 
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related to the expenditure of $100, in the interval $2,400 to $2,500, 
unless he has also agreed to purchase the first $2,400 of the activity* s 
production. 



C ase 9 

The final case considers the evaluation of Cases 7 and 8. It is 
the general problem faced by the decision-maker who has an ongoing 
system of activities and wants to evaluate current operations so as to 
prescribe subsequent action. 

The evaluation of Case 7 (no constraint on resources) is similar 
to Cases 5 and 3 and is not repeated here. The one difference is that 
the system of Case 7 has multiple objectives and therefore one would 
encounte more of the same comparisons. 

The evaluation of Case 8 leads to the central purpose of this 
dissertation which is to develop a generalized program evaluation model 
for ongoing educational systems. While the system being developed is 
capable of many types of evaluation, the principal thrust is to evaluate 
those aspects of the system necessary to improve the allocation of the 
next period budget. 

The structure of the analysis leans heavily on the concepts and 
constructs of Cases 7 and 8. The overall system objectives and values 
provide the production target. What information is required in order 
to determine how well the system produced? In an overall sense it is 
the sum of the production (effectiveness) scores for the various 
activities. This total relative to the maximum score possible rates 
the system in terms of relative effectiveness. That is. 
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REL (E*) 




, IT j * 1 in the denominator, 

^ rt% t 




R 



•l = A 



£ v{Ojj) 



The relative productivity of the system toward overall system 



objective A is 



EEL (E*) ' » 




, V(0p>0. 



These overall evaluations do not provide the basic information 
necessary to evaluate for the purpose of making future-period decisions. 
Presumably, decisions made in the present for the next period should 
incorporate the same kind of analvsis as was used in Case 8; that is, 
a systematic search over the alternative possibilities so as to arrive 
at a combination of activity levels which will lead to a better produc- 
tivity. 

This calls for an assessment of the cost-effectiveness relationship 
for each ongoing activity. The assumption is that activities will be 
carried forward although this is not a necessary assumption for the 
analysis.. The reason for this assumption is to avoid the consideration 
of alternative activities in the present context. A cost-effectiveness 
curve is developed for each activity by borrowing the curve used to 
plan the activity in Case 7 and then presenting the relationship to the 
implementation staff. 



I 
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Figure 9 shows a cost-effectiveness curve for activity a^. It 
represents a curve that has been revised from the £ priori relationship 
presupposed by the analysis of Case 8. Several aspects of the relation- 
ship warrant attention. With regard to costs C*^^is the amount of 



Effectiveness (Ejjjj;) 



Figure 9. Revised Cost-Effectiveness Curve. 




Cost (Cmi) 












resources actually expended and the level allocated to the activity 
to pursue overall objective If the activity were a developmental 

activity , then the expenditure of Cjj " additional dollars would 

allow productivity to rise to (assuming the adequacy of the revised 

activity were, on the other hand, concerned 
with the maintainance of an ongoing activity , then it would require Cx 
dollars to produce at a level of 11^. Since the purpose of this analysis 
is to evaluate ongoing educational systems, the developmental activity 



^Practically, probably be close to zero since 

administrators are reluctant to return unexpended resources for obvious 
reasons. The firm that can find ways to make administrators comfortable 
with returning resources that cannot be spent wisely will benefit 
substantially. This problem is probably more acute in public sector 
organizations. 
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problem Is disregarded. This does not imply that schools do not engage 



in project type activities. Obviously,, the methods developed here can 
be extended to the whole class of ^ hoc activities, but the emphasis 



should relate to ongoing activities and programs. 

An inspection of the cost-effectiveness curve such as the one in 

^ * * 2 

Figure 9 yields a point ,E^ < representing the level of productivity 
during the current year. It is intuitively obvious that the implementa- 



tion staff has a reasonably good idea of how production would be 

* 

Increased if there were small changes in cost from the level of the 
current year's expenditures. They are less certain of the response for 



large changes from the present expenditure level. The engineer on 



the other hand has studied the production function and consequently 



is more aware of the process that relates the inputs and outputs than 
the implementation staff. Consequently, it would appear to be a good 
idea to have the engineer available for consultation (if not direct 
involvement) In the process of assessing what happens if costs are 
varied in light of the present period experience. The difficulty is 



that the responsibility for subsequent implementation and maintenance 
of the activity is remote from the responsibilities of the engineer. 



This means that those responsible for these evaluations will probably 
not involve the engineer. 



At any rate, the Implementation staff should prepare a schedule 

* 



that lists expenditure levels above and below well as the estlma- 
ted responses in production associated with the changes from These 
are analogous to the dE^^* values generated in Case 8. 
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The same type of search process is instituted as in Case 8. There 
is, however, the problem of what will be referred to as human factors, 

H. If resources are to be taken away from one activity and given to 
another based on cost-effectiveness considerations, then the decision- 
maker is painfully aware of such human elements as loss of morale. While 



no attempt is made to study or measure H, it does figure into the 

A^nM(t+i) 



-HriA 

then resources will be diverted from activity to activity a^. In 

this inequality the symbols are defined as 

change in performance in year t+1 associated with 
'■ ^ the cost increment if it would be added to a 

XDJk 

= the decision-maker's assignment of potential value to 
activity a^as it would contribute to overall objec- 
tive 4 . 

= the increment of resources that is being considered 
for transfer from Sj^^^to 

*= a dollar unit assessment of the human factors associ- 
ated with the loss of ACX... dollars by the admin- 
istrator of activity a^^^ (t+1) and the gain of 

these dollars by the administrator of activity a • . 






decision process. In general, if 






B. Summary 

In summary. Case 9 has drawn from the other cases in order to 
provide a general program evaluation model for ongoing school systems. 
Case 1 provided a starting point in terms of certainty equivalence and 
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generated a criterion for selecting an activity aimed at one objective 
with no constraining factors. Case 2 instituted a cost-constraint and 
shov;ed that the only result was the possible elimination of the preferred 
activity of Case 1 because it costs too much. Case 3 indicated that 
evaluation of Cases 1 ard 2 was not essentially a statistical problem. 

It also stressed the purpose of evaluation as improvement of subsequent 
operations. Case 4 considered multiple activities aimed at the one 
overall objective. The need for a weighting of outcomes according to 
their worth to the decision-maker was indicated. The resulting criterion 
was a weighted performance. called expected worth. Case 5 brought in costs 
and the results were similar to those of Case 2. The implication of a 
cost constraint was that some packages could be eliminated from consid- 
eration, but if the decision-maker had only a single objective there 
was not a problem of cost versus effectiveness. 

The evaluation of Cases 4 and 5 in Case f was similar to Case 3. 

Again the emphasis was placed on evaluation for future improvement in 
operations. Case 7 introduced a set of overall system objectives and 
considered how to select a set of activities tc produce the objectives. 
Productivity was defined as a function of performance and potential 
value. Case 8 provided a cost-effectiveness framework to decide among 
the proposed activities designed to produce the overall system 
objectives. It was seen that a decision-maker only has a cost-effective- 
ness problem when he pursues at least two objectives and has a level 
of resources that restricts at least some of the activities that could 
be conducted. Case 9 ties the cost-productivity relationships of the 
current year to the budgetary needs of the next year. The central 
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feature of Case 9 is the cost-effeetiveness function. It was indicated 
in Chapter I that educators seem to know very little about the processes 
that take the inputs of education and link them to educational productiv- 
ity. 




III. APPLYING THE MODEL TO A SCHOOL SUB-SYSTEM 



This chapter uses the evaluative structure of Case 9 and applies 
it to the programs of a sub- system of an ongoing school system. There 
are two main reasons fcr applying this model to a school system sub- 
system. Since it is unrealistic to assume that existing school infor- 
mation systems are capable of providing relevant data to support 
managerial decisions with regard to the allocation of resources, the 
first priority is an assessment of the practicality of obtaining the 
data required by the model from school sys terns. The second priority 
is also pragmatic and is a determination of the extemt to which the 
type of analysis suggested by the model is useful to the school decision- 
maker. 

Basically this chapter presents background information about the 
pilot school ^stem and then considers the sequence of steps adopted to 
apply the model and provide data. The next chapter uses these data to 
generate and discuss model outputs. 

A. Selection of the Pilot School Sub-System 
The Radnor Township School District is located approximately 



should be pointed out that numerous efforts are under way to 
design management information systems for schools. Most of these plans 
start with a premise that since school decision-makers do not kr.ow what 
they want, the planners will design a flexible system that can relate 

*hll possible data** in ’’all possible ways." One project , previously refer- 

red to started vTith a set of questions , structured from the school admin- 
istrator's point of view, and proceeded to develop a PPBS for intermediate 
units and school districts in Pennsylvania* * This, however, seems to be an 
exception to the prevailing "flexible design in the face of ignorance*' 
approach. 
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twenty-five miles from the center of Philadelphia and is clearly sub- 
urban by virtually any socio-economic criterion. The school system 
operates a senior high, a junior high and four elementary schools. The 
pupil population is in excess of 4.200, with approximately 950 students 
in the senior high (grades 10-12). The 1967-68 proposed budget for 
the school district was slightly more than $3.8 million. 

The major reason for selecting this school system for study was that 
it had a strong reputation for being progressive, especially with regard 
to self evaluation and improvement. The system is small and consequently 
the prospects of keeping data collection problems under control enpeared 
to be good. Once an initial contact was made with Radnor administrators, 
a series of alternatives for possible study were considered. The 
health and physical education administrators seemed interested in the 
study and were very cooperative. In addition, health and physical 
education programs seemed to eliminate the problems associated with 
studying learning outcomes. While these problems should be treated in 
an overall school program evaluation effort, learning outcomes present 
a host of difficulties. In this limited study it was thought best to 
(jij^cumvent these problems. An additional advantage vras that the student 
population, program staff, and records were housed in the same building. 
Finally, it was decided to restrict the effort to the health and ihysical 
education programs in the Senior High School for school year 1967-68. 

The author identified three persons as being critical to an 
analysis of the operation of the sub-system: the Assistant Superinten- 
dent for Curriculum, who is ultimately responsible for planning and 
decision making in the area of health and physical education, the 
Physical Education Coordinator for the school system, who is also the 
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Chalrman of the Physical Education Department in the Senior High School, 
and the Nursing Coordinator for the school system, who is also the School 
Nurse in the Senior High School. Both coordinators report to the 
Assistant Superintendent for their system-wide responsibilities, but 
at the same time are responsible to the Senior High School Principal 
for the high school programs. The coordinators participate in system- 
wide scheduling and budgeting processes while the Principal makes Senior 
High School budget decisions. Obviously, a great deal of cooperation 
and communication is required. 

® • Application of the Model 

The purpose of this section is to describe how the model was 
applied to a sub -system of a school system in order to evaluate the 
contribution of programs to the productivity of the system. It is 
important to maintain the focus of the analysis on evaluation as a 
means for improvement of ongoing systems as opposed to an optimization 
methodology. 

A distinct preference for system-wide evaluation as an approach to 

improving schools is advanced by Burkhead. He contents himself with 

"analyzing relative costs and gains, not for an optimum allocation 

pattern" in his input -output study. This compromise is founded on 

the many technical difficulties and complex socio-economic interrelation- 

4*6 

ships which inhibit attempts to control important variables. 

A similar point of view in terms of improving resource allocation 
in education is expounded in a study by Abt Associates. They indicate 
that the goal is optimal decisions "but optimization is much more 
difficult than evaluation, since it involves the generation of programs 

46 

Burkhead, Fox and Holland, op. cit. , pp. 10-13. 
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vhereas evaluation does not. Optimization is a sensitive complex 
problem, but if only evaluation can be carried out, this alone will be 
useful. 

The remaining portions of this chapter describe the implementation 
of the model as an evaluative structure. The resulting structure 'rc- 
vides the linkage between Case 9 and the health and physical education 
activities of Radnor. A plan describing the sequence of activities 
required to implement the model is presented in Exhibit 3. Included 
with the implementation functions are several application steps as 
well as a set of steps describing the relationship between Case 9 and 
Case 8 for the next period. It is important for the analyst to 
familiarize himself with the activities and style of operation prior to 
undertaking the implementation of the mode lementation steps 

are: 



1. Define overall objectives. 

2, Assign values to overall objectives. 

3, Reconcile activities to overall objectives. 

4. Distribute value to activities. 

5. Define performance criterion for each activity. 

6, Define cost criterion for each activity. 

These steps permit the basic structure to exist but are Incapable 
of generating evaluative outputs. In order to apply the model, two 
application steps are necessary to feed the relevant data through the 
structure: 



^^Abt Associates, op. cit. , p. 1, 
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1. Observe perfonaance outcomes for each activity. 

2. Observe cost for each activity. 

The next section describes the familiarization efforts previously 
mentioned. 

1. A prior step 

A prior condition for evaluation is that activities are being 
conducted. Usually these activities are conducted under the headings 
of programs, the definition of a program being arbitrary. A program 
is defined, for purposes of this study, as a set of activities which 
are related. Often the logic of the relationship is administra- 
tive; that is, the activities are administered by one person and are 
grouped under a program label solely on this basis. 

The activities at- Radnor in the "health and physical education" 

48 

domain were grouped under the descriptors that follow. This structure 
was jointly agreed upon by the Assistant Superintendent for Curriculum 
and the author during an interview on February 23, 1968. 



HealthrServices Programs 

1 . Annual Screening 

2. Physical Examination 

3. Communicable and Infectious 
Disease 

4. Emergency 

5. Follow-up 



Physical Education Programs 

6. Physical Education 

7 . Adapted Physical Education 

8. Health Education 



48 

Appendix B describes the eight programs as they operated in 
school-year 1967-68. The grouping of some activities in this structure 
is arbitrary and in some cases misleading. As the study progressed activ- 
ities were shifted so as to provide a better description of school reality. 
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EXHIBIT 3 



AH OUTLINE OF THE CASE 9 EVALUATION MODEL AND ITS RELATIONSHIP TO 

THE CASE 8 ALLOCATION MODEL 



fi)EFINE overall! CdEFINE I .BeFIHE PERFORMANCE 
OBJECTIVES I I ACTIVITIES Hjc 

II I ACTIVITY^ 

ASSIGN VALUESi ^ RECONCILE 
CO OVERALL L v ACTIVITIES! 

)BJECnVES_p^ TO OVERALL 






DEFINE COST C 

RITERION FOR EACHl IfOR EACH ACTIVITY 



ite^onI 



OBJECTIVES 



)ISTRIBUTE 
VALUE TO 
ACTIVITIES 



lOBSERVE PERFORMANCE! 
OUTCOMES 



DETERMINE 
. EFFECTIVENESS 
■ FOR EACH 
ACTIVITY 



OBSERVE 
COST FOR 
EACH 
ACTIVITY 



CONSIDER NEW 
ASSIGNMENT OF 
VALUES TO OVER 
ALL OBJECTIVES 

DETERMINE NEW 
BUDGET LEVEL 



IGENERATE CASE 9[K 


[SUGGEST CHANGES 


[evaluation 0UT“L 


Jbased on evalu- 


1 PUTS 


~ ATION 




I 



^DEFINE COST 
EFFECTIVENESS CURVES 
toR EACH ACTIVITY 




1 




IJONSIDER DEPARTURES FROM 


PREVIOUS YEAR'S COST 




EFFECTIVENESS POINT FOR 


(each activity 














f w^Kwr^ 
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Once the author wf/.s able to deri.ve an understanding of the nature 
of the programs, the first formal task — defining overall health and 
physical education objectives-- was considered. 

2. Overall system oblectives 

Just as firms produce to meet consumer needs (and thereby enjoy 
profits), school systems produce to meet pupil needs. A utility 
function could be postulated to express each pupil’s need for various 
amounts of the types of services that the school produces, given the 
amount of each service he already possesses. The individual will have 
lis utility maximized when the ratios of the marginal utilities for 
the respective goods are proportional to the amounts of the constraining 
element used. Ordinarily the prices of the commodities are related to 
the marginal utilities to determine if the consumer's position can be 
improved. In this instance, prices are not relevant. Since time would 
be a major factor limiting an individual's consumption of various amounts 
and combinations of educational commodities, it would function as a 
conptraint . 

A relationship for each pupil could result from viewing the pupil 
as a consumer of economic goods. The function describes individual 
I's utility for the various educational services. There is a total of 
T time available. Individual 1 requires t^j^^ of time to consume a unit 
of educational service The expression 3 u^ is the individual's 

gxif 

marginal utility for a unit of Xa. 
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Th6 function is to bo maximized subject to the constraint 
expression. 

Zi-’Jj(Xa,Xb, .... X^) +>FjTi-(taXa + + ... + t„Xt,)] 



o ^(Fj) = 



iili -Xt. 



* 0 



dX 



ai 



^ ^ ^(Fj) * ^^1 “ ® 

dXb^ dXbi ^Xbi 



^ dui o a(Fi) .JUL. " ^ 

" dXni 

A - ^1 = ^ s . . . = iui 

/taj '/tb, V 

The conclusion is that the consumer (student) will maximize his 
utility when the marginal utility of the respective educational services 
is proportional to the incremental time across all services. From these 
relationships it follows that in order to have the school administration 
provide optimally for a pupil, it is necessary for him to know: 

1. How much of each educational service the pupil "needs". This 
could be determined by estimating how much of the educational 
service he "has" and how much he "should have". The utilities 
could be assigned by a knowledgeable school counselor. 

2. Once needs are assessed it ir necessary to describe the function 
which relates the pupil’s consumption of the educational service 
to the time it takes to affect this consumption. This would, for 
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example, be the learnir.g curve in instances such as arithmetic 
and reading. 

From this type of schedule the school administrator could sum over 
the pupil population and obtain estimates of how much of the various 
educational services should be produced and for whom. The present discus 
sion i3 not an attempt to minimize the problems associated with a 
meaningful assessment of pupil needs. First, it is presumptuous to 
assume that the child knows what he needs or that he would necessarily 
select what is best for him, if it were offered. Allowing the parent 
to serve as a proxy consumer would probably give some limited improvement 
Oue approach to the assessment of pupil needs would involve a survey 
or inventory of needs with the information interpreted by the decision- 
maker (and his expert advisors). This interpretive role is consistent 
with present school practice since it is implicitly part of the budget 
allocation process. 

The approach used in this study assumed that the Assistant Super- 
intendent for Curricultim has the responsibility for interpreting pupil 
needs or at least reviewing the interpretations of the two coordinators. 

A set of overall health a^* " physical education objectives were developed. 
This was not an easy task. A IStudy by Lawrence Downey indicates that 
educators faced with the job of trying to spell out the objectives of 

education were often in substantial disagreement. He cites instances of 

49 

outputs such as 'unified view of self in the universe' . 

49 

'Lawrence W. Downey, The Task of Public Education , Midwest Adminis- 
tration Center (Chicago; The University of Chicago, 1960), p. 15. 
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Dyer, in discussing the problem of objectives in connection with 
how educators allocate resources says: 

The trouble is that in spite of all the hard thinking 
and earnest talk about educational goals and how to define 
them, the goals produced have been essentially nonfunctional -- 
and I mean even when they have come clothed in the so-called 
behavioral terms we so much admire. They have had little or 
no effect on the deals and deliberations that go on in 
faculties and school boards and boards of trustees and legis- 
lative chambers where the little and big decisions about 
education are being made. As you watch the educational 
enterprise going through its interminable routines, it is 
hard to avoid the impression that the whole affair is mostly 
a complicated ritual in which the vast majority of part icipants- 
pupi Is, teachers, administrators, policy makers - have never 
given a thought to the question why , in a fundamental sense, 
they are going through the motions they think of as education?® 

The overall health and physical education objectives were defined 



by the Assistant Superintendent and the author during an interview on 



March 22, 1968. 


They were: 




Symbol 


Label 


Description 


°A 


Appraisal 


Every child should have his 
health status appraised in 
many respects. 




Follow-up 


Possible faults, defects, etc. 
require follow-up. 


°C 


Emergency 


Health emergency situeitions 
should be provided for adequate- 
ly. 


% 


Health Education 


Health education is to be 
provided . 


Oj; 


Physical Education 


Physical Education is to be 
provided . 



The author maintains that the purpose of overall objectives is to 



^®Henry S. Dyer, "The Discovery and Development of Educational 
Goals," Proceedings of the 1966 Invitational Conference on Testing 
Problems , New York,October ,1966, (Prince ton; The Educational Testing 
Service), p, 13. 
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provlde a foundation on which it is practical to categorize specific, 
observable activities. Specificity occurs at the activity level, with 
each overall objective serving as a framework within which one is 
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target area. The criterion of adequacy of these overall objectives 
is indeed pragmatic — l.e., the ease with which the activities to be 
evaluated can be distributed to the overall objectives by the decision- 
maker. In this study, the five overall health and physical education 
objectives were adequate. Initially, there was a sixth overall objective, 
"Communicable Disease", but the decision-maker merged it with "Emergency". 

3. Overall system objective valuation 

The need to assign values to objectives of differing importance is 
not perceived clearly in the educational administration. Recently, 
the Pennsylvania Study Commission tried to structure school system 
objectives. The approach involved a panel of educational and civic 
leaders and experts. There was reluctance to assign weights, as 
measures of importance, to the objectives generated on the grounds 
that "all ten were important goals of education and that none should 
be neglected" . 

The need for weighting objectives is obvious (although politically 
it may not always be wise) , since if they are not differentiated the 
implicit assumption is that they are of equal importance. Rankings may 
provide helpful information, but are not very useful in most decision 



Plan for Evaluating the Quality of Educational Programs in 
Pennsylvania. Vol. 1; Basic Program . A Report from the Educational Tes- 
ting Service to the State Board of Education (June 30, 1965), p. 11. 
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situations, since questions arise such as — "do I produce the 
objective ranked first and disregard the others, or.,." 

The method used in this study to weight objectives is referred 
to as the Churchman - Ackoff Objective Weighting Model, The reasons 
for selecting this approach were that the method is relatively simple 
and provides a measure of relative weights for the decision-maker. Also, 
most of the other valuation and scaling techniques require multiple 
judges, and in this study the emphasis is the single designated decision- 
maker who had the overall responsibility. 

The method provides a mechanism for weighting multiple objectives 
on the basis of repeated comparisons and produces a set of internally 
consistent weights. It does this by allowing the rater to value and 
revalue when inconsistencies arise. The assumptions of the method are: 
"1. For every outcome 0 j , there' corresponds a real non-negative 



number V^, to be interpreted as a measure of the true importance 



of Oj. 



2. If Oj is more important than Oj^, then Vj>Vj^ and if 0^ and Oj^ 
are equally important, then Vj ~ Vj^. 

3. If Vj and correspond to 0^ and respectively, then V.+Vj^ 

corresponds to the combined outcome 0^- and -0, 

J K 



The first statement postulates the existence of a vrt.ue function. 
The second is that of transitivity and the third specifies additivity. 

In allowing the Assistant Superintendent for Curriculum (the 
decision-maker) to apply this method to objectives, there were several 



West Churchman, Russel L. Ackoff, and Leonard E. Arnoff, Intro- 
duction to Operations Research (New York: John Wilev & Sons, Inc. ,1959), 
p. 140. 
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problems. The method is designed to force the rater to assess and 
reassess objectives singularly and in combination and, ideally, 
provides a set of weights with internal consistency. The decision- 

uia EVCX. WOO LTJ. XOJLOVS G^U\/\X\* waste pusA.KVV0Wc9 vr^ w»*w *Uw O— • — — — — — 



set of directions to serve as a guide during the interview. An 
unexpected problem arose due to the fact that the decision-maker felt 
that the objectives in some groupings were of nearly equal Importance. 
These perceptions restricted the nature of the comparisons he was able 
to make. For example, in a typical experiment, cited by Ackoff, et.al., 
the preliminary weights assigned were 1.00, 0.80, 0,50 and 0.30.^^ It 
is far easier to combine these quantities than a set of initial weights 
such as 1.10, 1.05, 1.00, and 0.90, yet this is the way some sets of 
activities were perceived by the decision-maker. 

Value problems are discussed in g;reater detail later in the chapter 
since the present section related to the valuation of overall health 
and physical education objectives. 

The values assigned were: 

Overall Objective(Oj^) ^(®a) 






110 



^^This comment refers to the fact that program activities were 
grouped so that they could be valued in terms of potential contribution 
to the overall objectives they were designed to produce. 

^^Churchman, Ackoff, and Arnoff, ov>. cit . , p. 139. 

^^Value assignments were made by the Assistant Superintendent for 
Curriculum during an Interview conducted by the author on April 30,1968. 



0 






0 , 






too 

140 

125 

125 



R 

jr 600 
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The critical assumption, given the internal consistency of the 

values to the decision-maker, is that the empirically derived values 

represent locations on a relative scale reflecting the prefei nces of 
56 

the rater. In a recent article, Robert Winkler discusses the reason- 
ableness of the assumptions underlying the measurement of subjective 
probabil i t ies 

Of course, the theory of personalistic probability and the 
underlying principles of coherence constitute a normative 
theory and do not claim to describe actual behavior. We 
would no more expect people to never violate the postulates 
of coherence than we would e:raect them to never violate the 
rules of logic or arithmetic £he cites Bruno de Finetti for 
thisj. Presumably, the degree to which an assessor obeys the 
rules of coher»^nce would depend on such variables as the 
familiarity of the assessor with the terminology of prob- 
s^bility and statistics and his general competence in quan- 
titative reasoning.^' 

The weights for the overall health and physical education objectives 
were obtained in April, 1968. As the study progressed there was some 



In Wroe Alderson and Paul E. Green, Planning and Problem Solving 
In Marketing (Homewood, Illinois: Richard D. Irwin, Inc., 1964), P.152, and 
Churchman, Ackoff and Arnoff, op. cit . ,pp. 150-152, there are suggestions 
that the quantities d -ived from the Churchman - Ackoff method can be 
combined with subjective probabilities to arrive at expected value 
determinations . 

Robert L. Winkler, "The Quantification of Judgement : Some Method- 
ological Suggestions", Journal of the American Statistical Association . 
Vol. 62, No. 320, (December, 1967), p. 1113. ' 
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uneasiness as to what these values represented. In March, 1969, it was 

decided to try to answer the question -- how internally consistent is 

this decision-maker in providing judgments for his overall health and 

physical education objectives? Since it was possible that his values 

could have changed substantially from the previous year, it was 

assumed that if he were consistent in 1969, it would be reasonable to 

assume that he was consistent in 1968. Accordingly, a test for internal 

58 

consistency was designed. 

The primary objective of this section is to determine the extent 
to which the Assistant Superintendent for Curriculum is consistent in 
his assignment of values. The procedure consisted of presenting the 20 
permutations of his five overall health and physical education objectives 

taken two at a time. The task was specified in the following manner: 
’‘Consider overall objectives A and B. Let A he valued at 1.0. How do you 
rate the value of B as a multiple of A?” An example was provided so that 
he was forced to think in a multiplicative sense rather than an additive 
sense. 

Before a.. 3 lycis of the data could be conducted, "t was necessary to 
define an aggregative statistic and to structure the problem mathemati- 
cally. The logic and development of this procedure are. given below. 

There are unknown parameters V^, V®, ... , A subject 

is asked to consider two of these parameters and Vg and to express ’’the 
value of B as a multiple of the value of A, just as if the value of A was 
set at 1.” 



^®The data were collected by the author during an interview with the 
Assistant Superintendent for Curriculum on May 12, 1969. These data are 
used later in this chapter to test the hypothesis of consistency. 
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The statistic x is a function of the subject's perceptions of and 

V|| and is the quotxeiit of the random variables^l^ and /Wa These variables 

are assumed to be independently and lognormally distributed. The latter 

assumption means the logarithms of the variates are normal and the 

variate itself is skewed positively with all values greater than zero. 

Aitchison and Brown indicate that the lognormal distribution has been 

used as a model to describe quantitative psychological responses to 
59 

stimuli. The independence assumrtion refers to the independence of 
errors in the subject's perceptions about V^j^. 



(The t appears in the denominator of the integrand since d(ln t) =l/t. ) 



This expression is the lognormal probability density and is given in 

61 

Aitchison and Brown. 

_ 

Aitchison J. and J. A. C. Brown, The Lognoimal Distribution , (Cam- 
bridge: Cambridge University Press, 1966), pp. 66-67. 



If a variable p is lognormally distributed, P 

is normally distributed, / The general form of a normal cumulative 

distribution 1 





£"£r* ^ 



and 








The notatio 

the lognormal dens .1 

in this section. 

Aitchison and J.A.C. Brown, op. cit . , 




The mean, for instance of 
.11 be discussed in more detail 



p. 8. 



ERIC 
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er|c 






<!•-> -> S V _ ■! 
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Altchlson and Brown give the results for the mean and variance but it 
is worthwhile for us to derive the mean since tne result is of consider* 



able Interest. The mean is defined as 



E(p) 



■1 



pf (p)dp. 



(- 2^<^"P >^P- 

Let Z * . then dZ = — n .* — dp, dp *#^/^.pdZ, and 

0i/7~ P 



E(p)= 1 









^ ^jTipdZ. 



fir- 

To eliminate p, let Inp ."Z-j/t; then p = e ^and 

=<P> ‘AT • 



The next step is to complete the square in Z: 

^ /T ■J|l+df2/2 dZ 



E(p) = 



- 1 

J9 






let t = Z - i then dt « dZ, and 



E(p) 



1 )Tt2 at ^'2 l(2)rM. 






Finally, E(p) * (denoted asSCby Altchlson and Brown) . 

This result is interesting since it means the expected value of a 
«Apvariate is a function o^but is not equal to^. The symbol^denotes the 
expected value of the density of In p. 



62 



illtchlson and Brown, loc. cit . 



1 

I 
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fin 

The variance of p is given as^ 

, where ;Si2 = ^ -1. 

Using these results for the distribution of and ^ we have 



i A 





'l!he expected values are 



= e^A+ •'i/2 , «e^ = e <4/2 



and variances are 




The need for a definition of an aggregative statistic that would 
test the hypothesis of consistency is apparent. Several alternative 



any time a rater assigned a perfectly consistent pair of values to two 
objectives the would be 0. 

The "’ery nature of the lognormal distribution was helpful in de- 
fining the statistic actually used to test the hypothesis of consistency. 
Aitchison and Brown discuss the nature of lognormal distributions; 



statistics were considered. The .statistic was defined as 



where 



and 




with, for example 




This statistic, R^, would present enormous practical difficulty since 



63 



Ibid. 
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The two-parameter lognormal distribution possesses a number of 
interesting reproductive properties, most of which are immediate 
consequences of those for the normal distribution. Since the 
latter has additive reproductive properties it is to be expected, 
from the characteristic property of the logarithmic function logXj^-f 
log X 2 = log Xj^X 2 , that the lognormal distribution will have 
multiplicative reproductive properties.®^ 

The actual statistic defined in this study to test the hypothesis 
of consistency takes advantage of the fundamental multiplicative nature 
of the lognormal distribution. The statistic R is defined as 



In order to apply the many results derived by Aitchison and Brown, 
the author assumed experimental Independence. This assxnnptlon is 



in random sequences. The importance of having the rater provide inde- 







based on having the rater estimate such quantities as and 



pendent assessments is stressed by Torgerson in a discussion of sub- 
jective estimation of ratios. At a later place in this section a more 



thorough treatment of statistical inder'endence is provided) 



64 



Ibid. , pp. 9-10. 



65 , 



Torgerson , p. 104. 
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A central limit theorem is cited by Altchison and Brown. 

^ sequence of independent, positive variates having the same 
probability distribution and such that E [logKj] 

and VAR^log 

^ n 

both exist, then the product ir Xj is asymptotically distributed as 

A ( »i ^ 2 . 66 

Another general theorem states that if is a sequence of ^-variates 

where X. is A (/f \ is a sequence of constants and c = e* is a 

j J 

positive constant, then provided ^bj AT j both converge, the 

product isACa f 

While the constants c and b^are not relevant to the problem, the 
theorem does provide that the product of lognormal variates is also lognormal 
with the new parametersylf-and^f^ equal to the sums of the respective 
means and variances. Since ln(l/X) =-lnX the distribution of 

isA(x^) =A(/4-/j[.op|+rB), 

and the distribution oX is 

This distribution of is, by virtue of the general theorem 

just cited, also aA-variate. It is defined by 

A (Sj)=A(^.^+ 4-^.2«*2+ 2,g) 

=A(0,2*^+ 2^). 



^^ Ibld. . p. 13. 
^^Ibld. . p. 11 . 



- 79 - 



The density of on the assumption that^ is 

c9 

The next step defines the probability density of R*, where R* = ^ S^. 

JL(R*) “A(C2 (0) ’ 

=JI(0. 4C" ff^), 

It has been established that the probability density of R* is 

The statistic R* is a product of values. When the 
rater is perfectly consistent for a pair of ratings the is 1.0. 
Departures from 1.0 are taken as evidence supporting the hypothesis of 
inconsistency. 

Since we know that the probability distribution of InR* is normal, 
^ (0,4C2^^)j it is convenient to treat R* in logarithmic form. Under 
the condition that VAR(lnR*) is known, the test for the hypothesis of 

consistency is structured as the standard normal deviate 

„ _ lnR*-0 
L ^ ~~~ • 

The variance of InR* was found to be hut is in terms of the 

standard deviation of the logarithms of any of the original variables, 

2 n 

such as/*/T. We want the maximum likelihood estimator of C 2 in order 

A 

to calculate the t-statistic from the sample evidence obtained during 
the interview of May 12, 1969. 

While the statistic R* is a product, the statistic InR^ is a sum 
and a correction factor is required to adjust for the fact that InR* is 
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not an arithmetic mean. The arithmetic mean of InR* is 



TnR* 



In In S2+ • • • + In 




.But In R* is C2 times InR*. 



In terms of the variance of these two terms we would obtain a relationship 
based on the fact that the variance of^ a constant multiplied by a variable! 'is 
the square of the constant multiplied by the variance of the variable, i.e, 
(C?T^ VAR( ^ InS.) = VAR^ S? InS. 1 . 

^ i=i " Vi=i /c^y 



Kcw we can structure the variance estimator of InR*. 



and 



VAR(lnR*) = 

/\ 

VAR(lnR*) * 



(C”TvAR(lnR*), 



”s2 



(cp 






The t-statistic is 

lnR *-0 




with degrees of freedom. 

Two sets of data were obtained from the Radnor decision-maker in 
order to test the hypothesis of consistency. Each of the sets was made 
up of the C” pairs of comparisons. The actual comparisons were presented 
in random order with one restriction — the permutations comprising a given 
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combination were not presented In adjacent order to the decision-maker. 
The data were: 



Set #1 



Prtmh-f nflt" nn v. O? X* 
"i 



Inx^or InxJ 



A/B 


1.1 


0.0953 


B/A 


0.8 


-0.2232 


A/C 


0.6 


-0.5108 


C/A 


1.4 


0.3365 


A/D 


1.0 


0.0000 


D/A 


1.0 


0.0000 


A/E 


1.1 


0.0953 


E/A 


0.9 


-0.1054 


B/C 


0.5 


-0.6932 


C/B 


1.7 


0.6306 


B/D 


l.l 


0.0954 


D/B 


0.8 


-0.2232 


b/e 


1.0 


0.0000 


E/B 


0.9 


-0.1054 


C/D 


1.4 


0.3365 


D/C 


0.5 


-0.6932 


C/E 


1,7 


0.5306 


E/C 


0.7 


-0.3567 


D/E 


0.9 


-0.1054 


E/D 


1.2 


0.1823 


In order 


to test 


the comp 



> 

i 

} 

i 

} 

> 

> 



lnx,*+lnx! 

1 



-0.1279 



■0.1743 



0.0000 



- 0.0101 



-0.0626 



-0,1279 



-0.1054 



-0.3567 



0.1739 



0.0769 



(lnx 4 +lnx])‘ 

' ^ X 



0.01635841 



0.03038049 



0.00000000 



0.00010201 



0.00391876 



o.vjJ.OJ^54i 



0.01110916 



0.12723489 



0.03024121 



0.00591361 



a consistent set of preferences against the alternative of Inconsistency, 
the t-statlstic was calculated: 
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t= lnR^-0 




= -0,7141 

0.4602 

= -1.55 



A second trial similar to the first set of ratings was also obtained 



and these data were subjected to the same test. 
Set #2 



Combination x,- or x! Inx. or lnx| 
L ■ ^ L 



Inx^ + Inx!^ (lnx£ + Inx!)^ 



A/B 


1.0 


0.000 


b/a 


0.9. 


-0.1054 


A/C 


0.6 


-0.5108 


C/A 


1.5 


0.4055 


A/D 


1.2 


0.1823 


D/A 


0.9 


-0.1054 


A/E 


1.2 


0.1823 


E/A 


1.0 


0.0000 



} 

\ 



-0.1054 



-0,1053 



0.0769 



0.1823 



0.01110916 



0.01108809 



0.00591361 



0,0^323329 
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Combination 


X. or x! 
1 i 


InXj^ or 


Inx! 

1 


lnx£+lnxj^ 


(Inx^ Inxp^ 


B/C 


0.6 


-0.5108 




0.0198 


0.00039204 


C/B 


1.7 


0.5306 


J 






B/D 


1.1 


0.0953 


] 


0.1906 


0.03632836 


D/B 


1.1 


0.0953 








B/E 


0.9 


•0.1054 


\ 


-0.2108 


0.04443664 


E/B 


0.9 


-0.1054 


J 






C/D 


1.7 


0.5306 


\ 


0.1739 


0.03024121 


D/C 


0.7 


-0.3567 


J 






C/E 


1.6 


0.4700 


\ 


0.2468 


0.06091024 


E/C 


0.8 


-0.2232 


) 






D/E 


0.8 


-0.2232 


] 


-0.1279 


0.01635841 


E/D 


1.1 


0.0953 









t = 


0.3409 




7 10/9 J 


[’0.25001105 - (0.3409)^3 


= 0.3409 




0.527- 



« 0.65 

The original purpose for structuring R* was to investigate the 
consistency of the values assigned by the Assistant Superintendent for 
Curriculum to the overall health and physical education objectives. The 
test for consistency was based on the assumption of independence of 
combinations within an individual sample. This assumption should be 
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explored in more depth since the results cited from Aitchison and 
Brown explicitly require experimental independence. 

The model implied in the use of the statistic InR* to test the 
hypothesis of consistency of assigned values is now examined. For 
elements A and B the value responses may be expressed as 



since the middle terms have expected values of zero. If the expecta- 
tion of the error covariance elements is assumed to be zero, implying 
a normal density of the error product, then the argument focuses on 



uli, were incorporated into the procedures used to assess the preferences 
of the Assistant Superintendent for Curriculum. 

Guilford, in comntenting on an article treating the problem of ex- 
perimental independence, indicates that the investigators recommend that 
the observer should be allowed to maintain confidence that he is 



to be 




The mathematical expectation of the product variable is 




the relationship between subsequent prooucts, for 

example * 



4:c * 



Some references to the scaling and measurement literature are 

68 

appropriate. Torgerson's suggestions, including randomization of stim- 



68 



Torgerson, p. 104* 
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ob serving and not guessing. Guilford concludes that the advice is 

69 

sound and necessary for assuming independence. 

Luce, on the other hand, takes a different position with 

. j Po -iri/i-fnAf-oc writers general Iv 

respect co respouse — 1 --^^ ^ 

persist in making the assumption of independence in spite of con- 
trary evidence. While Luce acknowledges that no one has formulated 

response dependent models he argues for such developments in order 

70 

to justify current practice. 

The task at hand is to consider the actual experiment to gain in 
sight into the inherent problem of dependency. Earlier we had de- 



fined 


ci 




/?* 


'T’* - 








. where, for example. 






• ^ . at./ 








If the various product elements. 


e.g. > were independent 



then we would expect their errors to randomly distribute about their 

exp ced values. The mathematical expectation of R* under the null 

hypothesis of consistency and the assumption of independence was zero. 



p. Guilford, Psychometric Methods , 2nd edit., (New York: 

Me Graw-Hill Book Company, Xnc., 1954), p. 323. 

The article discussed by Guilford is: 

V. L. Senders and A. Sowards, "Analysis of Response Sequences 
in the Setting of a Psychological Experiment," American Journal 
of Psychology. Vol. 65, (1952). 

Duncan Luce, "Detection and Recognition," R. Duncan Luce, 
Robert R. Bush, and Eugene Galanter, Editors, Mathematical Psychology , 
(New York: John Wiley & Sons, Inc., 1963), p. 106. 
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Consequently, the test for the hypothesis of independence would 
involve a consideration of the deviations from E(lnR*). 

An ideal treatment of the problem would involve many values of 
InR*. The deviations from E(lnR*) would be tested for goodness of 
fit with the normal curve. 

In a more limited context the hypothesis 

could be tested « 

In summary, there does not appear to be an adequate method to test 
the hypothesis of independence. 

We can, however, examine the respective combinations of values 
across sample trials. Here the hypothesis of no combinations by trial 
interaction can be tested. The null hypothesis is 

"t-ix i-ix ) j 

where the JL subscript refers to the comb inat ion ^and 1 and 2 to 
trials. 

The test procedure is taken from Morrison and is the standard test 
for the significance of the difference between two random variables 



71 

A complete investigation into questions concerning population 
profiles is provided by Morrison. The method of "profile analysis" 
is extended to the multivariate case. 

Donald F. Morrison, Multivariate Statistical Methods , 

(Me Graw-Hill Book Company, 1967), pp. 141-148, 186-197. 






from dependent seinples. Th.e test statistic under ^0 i® distri 
buted as Student's t with^-| degrees of freedom. The statistic 

is 



A JF 

r s ^ 



A, - /L JL 









_ 



4a 



K 


Ik 4 












I'MiUi'iy 








Trial #1 


Trial #2 






• 

Combination (4 ) 
AB : BA 


-0.1279 






Jj 


-0.1054 


-0.0225 


0.0005 


AC : CA 


-0.1743 


-0.1053 


-0.0690 


0.0048 


AD : DA 


0.0000 


0.0769 


-0.0769 


0.0059 


AE : EA 


-0.0101 


0.1823 


-0.1924 


0.0370 


BC : CB 


-0.0626 


0.0198 


-0.0824 


0.0068 


BD : DB 


-0.1279 


0.1906 


-0.3185 


0.1014 


BE : EB 


-0.1054 


-0.2108 


0.1054 


0.0111 


CD : DC 


-0.3567 


0.1739 


-0.5306 


0.2815 


CE : EC 


0.1739 


0.2468 


-0.0729 


0.0053 


DE : ED 


0.0769 


-0.1279 


.0.2048 


0.0419 


-1.0550 


0.4962 


^ - -1.0550/10= 
SJl^* 0.4962 


-0:i055 til- 1 0.4962 1.1130 

'M I p ' 


= 0.21 




1,1130 


t« -o.iossji^o.ii = -1- 


58 





There is a fair amount of evidence to indicate that the hypothesis 
of no combination by trial interaction is dubious. This means that 
the responses for the various combinations appear to have interacted 
across sample trials. 



^^Morrison, pp. 133-134. 
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4. System activities and overall objectives 

Activities were studied by the Assistant Superintendent for Curricu- 
lum and the author and related to the five overall health and physical 
education objectives during an interview on April 30, 1968. This step 
was the most critical step in the sequence since it provided the basic 
linkage for the entire network. Activities are the fundamental elements 
of the system since values, performance scores, and costs are all related 
at the activity level. 

The eight programs previously listed were denoted P ,P«,...,P ,..., 

1 ^ m 

Pg. Activity J. of program m, aimed at overall objective was denoted 

Table 1 lists the 48 activities, by program, with a brief descrip- 
tion of each activity. 

5. System activity valuatio n 

Once the Assistant Supeiintendent for Curriculum had assigned a 
scale of educational value to overall objectives, he was able to relate 
health and physical education activities to these overall objectives. 

The author was then able to distribute the productivity potentials to 
the various health and physical education activities. Since activities 
were designed to contribute to specific overall objectives, they derived 
their potential for production from those overall objectives. Therefore, 
activities were not valued relative to other activities in their program. 
Within a given program the relatives can be on different scales until 



they are anchored by the unit of value assigned to the overall objectives. 







T'" ■ 
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TABLE 1 

PROGRAM ACTIVITY RECONCILED TO OVERALL OBJECTIVES , BY 
PROGRAM, WITH BRIEF DESCRIPTION 



Program Program Activity(a^) 

(P ' Reconciled to(Oj^) 

lU 


Brief 

Descri.pt ion 


Annual Screening 0^ 


vision appraisal 


®12 °A 


hearing appraisal 


®13 ®A 


height-we ight appraisal 


^14 


discuss problems with pupil 


a 14 Og 


discuss problems and health 
education topics with pupil 


®15 °B 


refer to parents 


"16 °B 


refer to teachers (other than 
physical education teachers) 


"17 °B 


refer to physical education 
teachers 


Physical Examination a2^ 0^ 


reflex appraisal 


a Oa 
22 A 


speech appraisal 


®23 


posture-orthopedic appraisal 


^24 ®A 


heart and blood pressure appraisal 


"25 °A 


other appraisal 


"26 °B 


refer to parents 


"27 °B 


refer to teachers (other than 
physical education teachers) 


"28 °B 


refer to physical education 
teachers 


®29 °B 


discuss problems with pupi? 


"29 % 


discuss problems and health 
education topics with pupil 
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TABLE 1 - Continued 



Program 


Program Activity 
Reconciled 


Brief 

Description 


Communicable and 
Infectious Disease 


"32 


education spillover 




^33 ^A 


detect venereal disease 




"34 °A 


detect tuberculosis 




^35 


refer to parents 




^36 °B 


refer to teachers (other than 
physical education teachers) 




"37 °B 


refer to physical education 
teacher 


NOTE: 


a 2]^ was transferred 


into another program 


Emergency 


^41 


care for serious injury 




^42 


care for less serious injury 




"43 ''c 


care for illness 




^44 % 


refer serious injury 




^45 


refer less serious injury 




46 B 


refer illness 


Follow' up 


"51 °B 


determine referral disposition 
when necessary 


Health Education 


"61 °D 


’’the pupil should accept respon- 
sibility for his own health" 




^62 °D 


the health curriculum 


Physical Education 


®71 °E 


every child should participate 




^72 ®E 


physical fitness 




«73 °E 


physical confidence 




®74 °E 


the program should present a wide 
variety of games and skills 




TABLE l"Ccntlnued 



Program Program Activity Brief 



Reconciled 

Physical Education ®75 °D 

®76 °D 

Adapted Physical ag^ 0^ 

Education 

"82 °A 
®83 °A 

"84 °E 



Description — 

proper attitudes 

prepare for lifetime sports 

posture orthopedic appraisal 

physical improvement 

proper attitudes toward physical 
problems 

children with physical problems 
should participate more 
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Distribution of value to activities was. accomplished by a simple 
procedure. Activities were grouped by overall objective and then sorted 
into sub-groups of five, six, and, in one instance,- seven. For example, 
twelve activities from five programs contributed io overall objective 0^ 
(Appraisal). They were divided into two sets (one containing six 
activities, the other containing seven). The extra activity was "posture 
and orthopedic," which served as a numeraire . 

During the course of the interview of April 30, 1968 certain factors 
became apparent. First, the superintendent seemed to have a set of 
values "internalized," In addition it was evident that he was not trying 
to rationalize the existence of the Radnor programs. He made numerous 
comments, such as "I am sure my teachers would put this number one, but 
my priorities are different." He agreed that the values obtained should 
be reassessed "in about a month" so as to indicate areas requiring 
obvious adjustment. 

Table 2 gives values assigned at the final interview a month later. 
The final values were obtained by showing the preliminary values to the 
superintendent and discussing possible inconsistencies and implications 
with him. 

Once the final valuations for the activities were obtained, they 
were used to distribute the potential value of the respective overall 
health and physical education objectives. This was done by summing the 
final valuation scores for each overall objective's activities and der- 
iving relative shares. Then the relative shares were multiplied by 
the total potential value for the relevant overall objective to yield 
standardized values. The maximum production possible for 



er|c 



obtained by; 



I 
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TABLE 2 

STANDARDIZED VALUES FOR PROGRAM ACTIVITIES, BY OVERALL OBJECTIVE 



Program Activity recondiled Final Relative Standa^ized Valne 
to: Appraisal (a^O^) Valuation 

mJ A / P»J A -j \j^bpjOA / 

Vl^mJ^A/ 





115 


0.09465 


10.4 


"11 


®12 


115 


0.09465 


10.4 


^24 


115 


0.09465 


10.4 


325 


115 


0.09465 


10.4 


313 


105 


0.08642 


9.5 


a 


105 


0.08642 


9.5 


21 




105 


0.08642 


9.5 


22 




100 


0.082304 


9.1 


“23 


381 


100 


0.082304 


9.1 




80 


0.06584 


7.2 


a 


80 


0.06584 


7.2 


34 


391 


80 


0.06584 


7.2 



1215 0.999988 109.9 



TABLE 2 -Continued 



Program Activity reconciled Final Relative Standardized 
to: Follow-Up Valuation Share Value 



^44 


250 


0.10965 


11.0 


^35 


150 


0.06579 


6.6 


^15 


150 


0.06579 


6.6 


^26 


150 


0.06579 


6.6 


^37 


125 


0.05482 


5.5 


^7 


125 


0.05482 


5.5 


"27 


125 


0.05482 


5.5 


"45 


125 


0.05482 


5.5 


"46 


125 


0.05482 


5.5 


"14 


125 


0.05482 


5.5 


^29 


125 


0.05482 


5.5 


"28 


125 


0.05482 


5.5 


^36 


120 


0.05263 


5.3 


^16 


120 


0.05263 


5.3 


^51 


120 


0.05263 


5.3 


^82 


110 


0.04824 


4.8 


^Q9 


no 


0.04824 


4.8 




2280 


0.99995 


100.3 


Program Activity reconciled 


Final 


Relative 


Standardized 


to: Emergency 


Va luation 


Share 


Va lue 


"41 


200 


0.47059 


65.9 


"42 


125 


0.29412 


41.2 


^43 


100 


0.23529 


32.9 




425 


1.00000 


•140.0 



1 
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TABLE 2 -Continued 



Program Activity reconciled 
to: Health Education 


Final 

Valuation 


Relative 

Share 


Standardized 

Value 


^14 


125 


0.13158 


16.4 


^29 


125 


0.13158 


16.4 


®61 


120 


0.12532 


15.8 


^62 


120 


0.12632 


15.8 


^75 


120 


0.12632 


15.8 


^76 


120 


0.12632 


15.8 


^83 


120 


0.12632 


15.8 


^32 


100 


0.10526 


13.2 


950 


1.00002 


125.0 


Program Activity reconciled 
to: Physical Education 


Final 

Valuation 


Relative 

Share 


Standardized 

Value 


^72 


100 


0.23256 


29.1 


^73 


90 


0.20930 


26.2 


^71 


85 


0.19767 


24.7 


^74 


80 


0.18605 


23.3 


00 

CO 


75 


0.17442 


21.8 


430 


1.00000 


125.1 



^er|c 
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where: 



b 0 /- the final valuation reflecting the relative potential 
^ of activity in any program producing 0 £. 



6. Performance Criteria 

Performance criteria are critical because they determine the extent 
to which the respective activities have done what they were designed to 
do. Chapter II referred to performance criteria in several ways. First 
there is the criterion itself, which prescribes what is to be observed, 
under what circumstances, etc. It is important to involve the decision- 
maker as much as possible in the process by which these criteria are 
selected. A second step in this operation is to develop the performance 
index This defines performance over its range (in this study 

the range is 0 to 1.0) and indicates exactly what has to be observed in 
order to assign an actual performance score, K*jO^. Generally, 
is a percentage or a rating of some kind. It is very likely that the 
superintendent, on learning that 90% of the children had their eyes 
examined, might say "yes, but 10% were not examined." He may feel that 
does not merit 0.9 . To account for the possibility 

that th'^ decision-maker might want to adjust performi-nce, provision is 
made so that he might transform into 

Before commenting on the nature of the performance criteria and 



performance outcomes at Radnor, it is apprpprlate to discuss some other 
aspects of performance criteria. In 1955 Aldersbn and Sessions, a 
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marke ting-management consulting corporation, devoted some space in their 

house organ to the problem of determining organizational effectiveness. 

They suggested a ’’figure of merit” as a measure of performance. This 

type of measure should meet two, sometimes competing, criteria. First, 

it should be acceptable to executives who are responsible for the system 

activities. Situations which they feel are preferred should obtain a 

higher merit score. The second criterion was that the index should 

. , 73 

be clearly defined and measuranle. 

The suggestion of using subjective appraisals when objective criteria 
Qxa not available is often made in the literature. The Abt Associates 
study comments on the necessity for subjective measures in allocating 
resources in educationj 

...Both costs and effectiveness theoretically can be measured, 
but there are many practical difficulties. At least costs can 
roughly be estimates in advance cn the basis of budget alloca- 
tions. The prediction of effectiveness is much more diftj-cult. 
Yet without some such prediction, however crude or subjective, 
there is no rational basis for deciding between one educational 
improvement program and another. 74 

Hitch and McKean, in the classic study of economics applied to 
nuclear defense considerations, point out that better subjective apprai- 
sals of both output and performance could be obtained by more meaningful 

75 

grouping of activities. 



^^Alderson and Sessions, Inc., ’’Systems and Models in Operations 
Research,” Cos t and Profit Outlook , Vol. VIII, No. 1, (Philadelphia. 
January, 1955), pp. 3-4. 

74 A 

Abt Associates, ibid . , p. 2. 

Charles J. Hitch and Roland N, McKean, The Economics of Defense 
In the Nuclear Age. (Cambridge: Harvard University Press, 1967), p. 61. 
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Both objective and subjective criteria were used to assess the 

extent to which activicies were produced in the Radnor case study. In 

some cases it was possible to observe performance or use counts as 

outcomes, while in other instances it was necessary for the author to 

elicit subjective assessments. In all, there were three interviews in 

which the author derived performance estimates from the Nursing and 

Physical Education Coordinators. These interviews were held in May of 

1968. Some of these judgments were based on data collected from pupil 

records and questionnaires. Appendix C considers the nature of each 

activity, the performance criterion used, the source of data, and any 

qualifying remarks by the Radnor participants. In quite a few instances 

the decision-maker qualified the subjective or objective rating assigned 

to a criterion. This amounted to the transformation of K*,0* to^*,0*. 

mj mJ « 

The first page of Appendix C discusses the method used to explain this 
transformation to the Assistant Superintendent. 

It should be emphasized that "performance" is not a comment on 
Radnor School District personnel, but is a quantitative rating assigned 
to an aspect of a program. In given instances, human performance can 
be of the highest caliber but'the level of program resources may not be 



A 



adequate to do the job. 

Table 3 presents the values of and K^Q^. It is of interest 

to note that in four instances the Assistant Superintendent elevated the 



score and in six instances he saw the need to lower se 

changes were made during an interview on May 16, 1968. A word of 
criticism about the study methodology is in order. While the criteria 
were developed before outcomes pbtained, the transfor- 

mation of K*tO-» into^* 0.* should also be done before the outcomes are 
X tnJ yt 



4 
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TABLE 3 



A 

ACTIVITY PERFORMANCE OUTCOMES K* 0 • AND ADJUSTED SCORES K* 0^* 

mJ >1 



A.nTTTrT'TV 

ex\j X X V X i. J. 



Vie f» • 



‘'mJ'"/, 



If* 0 • ACTIVITY K*,0/ 

"mj ^ ^ ^ 



K*,0/ 
oj ^ 



Oa 



Hi 

a 14 0 



^13 



St 



A 
B 

^14 % 
a 15 0 
a 



B 



B 



B 



16 
^17 
321 JJa 

Hi ®A 
a23 0^ 

324 °A 



^15 

®26 

327 

*28 

29 

®32 

a 



°B 

Ob 

S^> 

0? 



?? 0^ 



335 On 



B 



0.97 


0.97 


®36 ®B 


l.OO 


0.91 


0.91 


337 


1.00 


0.98 


0.98 


341 Oc 


1.00 


0.75 


0.75 


342 Oc 


1.00 


0.50 


0.50 


^43 Oc 


1.00 


0.20 


0.75 


^44 S 


1.00 


0.80 


0.80 


^45 S 


1.00 


0.40 


0.40 


®46 ®B 


1.00 


0.90 


0.90 


®51 % 


0.80 


0.90 


0.90 


^61 ®D 


0.75 


0.90 


0.90 


^62 % 


Not Rated 


0.91 


0.91 


Hi % 


0.99 


0.91 


0.91 


Hi ^E 


0.90 


0.17 


0.45 


®73 °E 
^74 ®E 


0.80 


1.00 


1.00 


1.00 


0.45 


0.45 


3-7C 0t> 


0.96 


1.00 


0.80 


0.70 


1.00 


0.80 


1.00 


0.25 


0.50 


®82 ®B 
®83 % 
®84 


0.40 


0.30 


0.30 


0.90 


0.95 


0.95 


0.75 


1.00 


1.00 


^92 °B 


Not Rated 
Not Rated 



1.00 

1.00 

1.00 

0.95 

0.90 

1.00 

1.00 

1.00 

0.95 

0.75 

0.40 

0.99 

0.80 

0.80 

0.85 

0.96 

0.55 

1.00 

0.40 

0.90 

0.75 

1.00 

0.00 
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TABLE 4 

ADJUSTED PERFORMANCE SCORES K*,0 • BY PROGRAM AND OVERALL 

mJ 4 , 

HEALTH AND PHYSICAL EDUCATION OBJECTIVE 

OVERALL OBJECTIVES 



Program Appraisal Follow-Up Emergency Health Physical 

(0.) (0„) (0-) Education Education 

“ (Oj,) (Og) 



Annual 11=0.97 

Screening 12=0.91 

(pp 13=0.98 



Physical 21*0.90 

Examination 22=0.90 
(P ) 23=0.90 

24=0.91 
25*0.91 

Communicable 33*0.30 
£c Infectious 34=0.95 
Disease (P 3 ) 

Emergency 

(P4) 

Follow-Up 

Health 

Education (Pg) 

Physical 
Education (Py) 



Adapted Physical 
Educ a t ion (Pg) 8 1«1 . 00 

Dental 

(P ) 

X 



14*0. 75 
15*0.75 
16*0.80 
17*0.40 




26*0.45 

27*1.00 

28-0.45 

29*0.80 




35*1.00 

36*1.00 

37*1.00 

44*1.00 


41*1.00 


4$*1.00 


42*0.95 


46*1.00 


43*0.90 


51*0.95 





82*0.40 



14*0.50 



29=0.80 



32=0.50 



61=0.75 

62*0.40 

75*0.96 71*0.99 

76*0.65 72*0.80 

73*0.80 
74*0.85 



83*0.90 84*0.75 



91*0.00 92-0.00 
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observed. This type of after-the-fact adjustment is not desirable since 

it gives a decision-maker the option to "make his system look good". 

Table 4 displays the values of K*_0 *by program and overall health 

mj m 

and physical education objective. 

7. Activity costs 

Costs in the case study were defined as "out of pocket" variable 
costs •“i.e. variable costs spent from the school budget. The rationale 
for considering only expenditures chargeable to the school budget is as 
follows. If the decision-maker could get volunteers to work at no cost 
to his school system, this could affect outputs. But it is entirely 
reasonable to regard a decision-maker as rational when he endeavors to 
provide gains for his operation "at no cost". From a societal point 
of view, he is suboptomizing since the same volunteers could perhaps 
produce more at a hospital. The main point is that the present effort 
is not attempting to assess the "real costs" of health and physical 
education or anything remotely related to this. Here the emphasis is on 
the school administrator as a rational manager. 

Jesse Burkhead draws the same conclusion when discussing Theodore 
Schultz's efforts aimed at evaluating investment in education in terms 
of a full range of benefits and costs to all segments of society. 

Burkhead says: 

A comprehensive cost-benefit approach is necessary for arriving at 
judgments as to whether we have, as a nation over - or under in- 
vested in secondary or college education. However, for the resource 
decisions that face a specific school board, it is necessary to look 
at only the benefits that accrue to the community, neglecting those 
that spill over to other areas, and to look only at the resource 
costs that are within the control of the board. 76 



Burkhead, Fox and Holland, op. cit ., p. 8. 
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The author expected that cost data could not be separated at the 
program activity level. Also, it was anticipated that sufficient infor- 
mation would be available so that estimates of 1967-68 expenditures bj 
program could be derived. Both expectations proved to be reasonable. 

There were several accounts which contained relevant cost data. 

These accounts were structured according to specifications which have 
been standardized in Pennsylvania school districts by the Pennsylvania 
Department of Public Instruction. 

1. Instruction 0200 

This includes salaries and supplies which relate to instruction. 

The Senior High School is denoted by 0222 with a budgeted amount of 
$436,097. Only a small proportion of this amount was separated for 
health and physical education salaries. Salary figures were aggregated 
at the program level in order to obscure the salaries of individuals. 

The four personnel involved in the administration and instruction of the 
physical education programs were paid a total of $36,800 in 1967-68. 
Exduded in the total were the athletic program and School Doctor salaries. 

2. Health Service 0400 

This includes salaries of nurses, doctors, dentists, and a dental 
hygientit . In addition, supplies and expenses are included. The total 
amount budgeted in 1967-68 was $63,975. Only a small segment of this can 
be tied to the Senior High School health programs. 

3. Procedures 

Cost Data were collected from the two major accounts. These 

expenditures were listed in the following manner* 

Description Vendor Amount ($) Date 

dixie cups Pontiac Co* 48.10 12 Sept. 67 
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In the case of the health service programs, the Nursing Coordinator 
assigned the various expenditures to the program which generated a need 
for the expenditure. Since many of the expenses were discrict wide they 
were pro rated on the basis of school population (unless more specific 
Information was available). The total public school enrollment was 
taken as 4388 and the Senior High School figure was set at 950. For 
example, the pro rata share of across the board supplies for the Senior 
High School was 950/4388 = 0.22. 

The physical education expense items were listed in the same fashion 
as the health expenditures. In this case the Physical Education Coordin- 
ator was able to match dollars to programs since the account for the 
Senior i.igh School was kept separately. 

The notion of amortizing equipment expenditures was discarded since 
the amount of money involved was small (less than $2,000 for all programs, 
spent on equipment during 1967-68) and the level of funds available for 
these purchases each year is somewhat uniform. 

In order to allocate the salaries to the various programs a time- 
study form was completed by the two Coordinators. The Nursing Coordina- 



tor distributed her time as follows: 

Program Percentage 

Annual Screening 30 

Physical Examination 5 

Cotanunicable and Infectious Disease 5 

Emergency and Illness 33 

Follow-up 2 

Other 25 

100 




Some of the estimates, for example, the Physical Examination and 

Poilow-up Programs, were figured on the number of school days required 

to stf.ff the program. These estimates were based on 200 school days. 

The Physical Coordinator split her time as follows: 

Program Percentage 

Physical Education 70 

Adapted Physical Education 25 

Other 5_ 

100 



In addition, she allocated 70 percent of the time of her three 
teachers to Physical Education and 30 percent to Adapted Physical 
Education. 

The School Doctor's time allocation was based on discussions with 
the Coordinators. Table 5 shows the total allocation of expenses and 
salaries to the various programs. As was mentioned previously, the 
data have been pooled. 

A few words should be said about the cost data. Due to the many 
problems associated with deriving the proceeding figures they should be 
viewed with caution. In Chapter VI recommendations are made concerning 
future ways to handle program expenditures and overhead costs. 

Some notation was defined so that subsequent analysis could 
facilitated. The total of expenditures for program m is 




Os 
mJ ^ 



and the expenditure associated with the production of overall objective 

p 

^ over all programs is 

8 
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TABLE 5 

SALARIES AND EXPENSES, BY PROGRAM 



ProRram 




Annual Screening 


2,961 


Physical Examination 


3,385 


Communicable and 
Infectious Disease 


913 


Emergency 


3,595 


Follow-up 


180 


Health Education 


3,000 


Physical Education 


27,259 


Adapted Physical 
Education 


11,825 


TOTAL 


53,118 



Expenses ($) Salaries ($) 



261 


2,700 


13 


3,372 


98 


815 


625 


2,970 


mm 


180 


* 


* 


1,499 


27,760 


206 


11,619 


2,702 


47,416 



* The $3000 Figure for Health Education is based on a proportion of a 
science teacher's salary and an estimate of cost of materials. 
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The total expenditure level for all programs is 



8 



8 




or 



R R 8 





C. Summary of Chapter III 



This chapter has examined how the model was applied to gather data 
in Radnor during school year 1967-68. In all, there were seven steps 
required. A prior step, study of the activities and programs at Radnor, 
was essential as a point of departure. Then five overall health and 
physical education objectives were devised and valued relative to each 
other. The next operation consisted of relating the health and physical 
education activities to the overall objectives. This anchoring process 
allowed the overall values to be distributed to the activities once 
the importance of the activities to each other (along production lines) 
was determined. Then performance outcomes and costs were obtained, 
the latter at the program level. 

There are several assumptions implicit in the way the data suggested 
by the model were derived and later manipulated. These assumptions are: 

1. The decision-maker has perfect knowledge of pupil needs and 
their importance to individuals. It is also assumed that this 
knowledge is reflected by the set of values attached to the 
overall objectives. 

2. The values assigned represent an internally consistent set of 



relative scores. 
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3.An appropriate definition of system output is value produced 
by the system. 

4. System activities are noninteracting. 

5. Production and costs are tied to the annual budget cycle. 

6. Efficiency for school systems is a desirable goal, and providing 
resource allocation procedures that try to foster production is 
a preferred way to accomplish this. 

In the next chapter the dsthor uses school year 1967-68 information 
collected at Radnor during the period February to May of 1968, to gener- 
ate outputs from the model. 




IV. MODEL OUTPUTS AND PRESENT YEAR EVALUATION 



This chapter utilizes data derived from the Radnor case study. 
Basically, evaluation outputs are of two types* A first kind attempts 
to satisfy the natural curiosity of administrators to find out how well 
their system did at the termination of the school year. Generally, 
these evaluations are at a higher level of aggregation, i.e. the program 
level and the overall objective level. The second Kind of evaluation 
is not designed to satisfy curiosity, but, rather, to present detail on 
cost-effectiveness relationships of activities. This flow of information 
represents one of the basic inputs for the budgeting process • 

In Chapter III it was stated that costs were not retrievable from 
the Radnor accounting system at the activity level. This meant the 
cost-effectiveness data obtained from school year 1967-68 vras at the 
program level. 

A 

Tabl^2 and 3 provided V(a^Oj) and values respectively. 

They are combined by multiplying each performance score by the maximum 
value attainable to form productivity or effectiveness outputs. The 
resulting products are the elements of Table 6 and are the basic inputs 
for touch of the treatment in this chapter. The logic of the evaluations 
presented in this chapter is basically a systematic movement from the 
most general outputs down to those at the activity level. 

Listed next is a description of the outputs directed toward 
answering the question - "How well did the system perform in 1967-68?". 
output description 

la. Overall system effectiveness, by overall objec- 

tive . 

108 



o 









- 109 - 



outputs 



description 



lb. 

ICf 



Overall system effectiveness, by program. 
Overall relative system effectiveness, by 



Overall relative system effectiveness, by 



overall objective. 



Id. 




Overall objective effectiveness, by program. 



Overall objective relative effectiveness, by 
program. 

Overall objective effectiveness, by activity. 
Program effectiveness, by activity. 

Overall objective relative effectiveness, by 
activity. 



Outputs grouped in sets 1 and 2 were aimed at providing assistance 
to the decision-maker in evaluating 1967-68 performance. Group 3 was 
primarily detail provided for those operating the programs. Detail of 
this nature is presented in Appendix D. It is interesting to note that 
the concept of "relative nfectiveness" at the activity level reduces to 

A % 

the performance score, ) since 



A. Overall System Effectiveness, by Overall Objec tive and Program, 
Overall system effectiveness is synonymous with overall system 
productivity and is the sum of the production elements across the system; 



3d. 



Program relative effectiveness, by activity. 



^a . 
4b. 
4c. 



Overall system effectiveness- cost ratio. 
Program effectiveness- ;ost ratios. 

The Elasticity of Cost-Effectiveness. 



5a. 



The 1967-68 cr st-effectiveness point, by 



program. 



and 



<j<>A ‘ ’ 

fv(a,u(^)] / V(anja^). 
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TABLE 6 

VALUE PRODUCED BY ACTIVITY, PROGRAM AND OVERALL OBJECTIVE 



0> 



0 



B 



Annua 1 
Screening 



Physical 

Examination 



Communicable 
and Infectious 
Disease 



Emergency 



^1 


10.1 


^21 


8.6 


^34 


6.8 


•12 


9.5 


^22 


8.6 


^3 


0.0 


%3 


9.3 


^3 


8.2 










^24 


9.5 












9.5 








28.9 




44.4 




6.8 


*^14 


4.1 


^26 


3.0 


^5 


6.6 


^15 


3.3 


^27 


5.5 


^36 


5.3 


^16 


4.2 


^8 


2.5 




5.5 


«17 


2.2 


^29 


4.4 








13.8 




15.4 




17.4 



^44 


11.0 


•45 


5.5 


^46 


5.5 




22.0 


A, 




41 


65.9 


•42 


39.1 


^3 


29.6 



8.2 



13.1 



6.6 



134.6 



Od 



14 8.2 



>9 13.1 



32 6.6 



°E 



50.9 



72.9 



■ 3 ^ 



T5^ 
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Follow-up Health Ed- Physical 

ucation Education 



Adapted Other Total 

Physical 

Education 




r\ 






-I 

J. 



91 



A A 
\J • 'y 



9.1 0.0 89.2 




(tsi 5. o' 




^2 . 0.0 



5.0 








1.9 


0.0 75.5 












134.6 




11.8 


^5 


15.2 


As 14.2 




^2 


6.3 


^5 


'6.1 








18:i 




23.9 


14.2 


84.1 


















(hi 


24.5 


^4 16.4 










23.3 










^3 


21.0 










^4 


17.5 












86.3 


16.4 


102.7 


5.0 


18.1 




110.2 


41.6 


” 0.0 486.1 
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or, alternatively, 



E = 



1 k I 

m=i m j a« ^ ^ 



= 486.1.- 



Uti/>n 1 nnlr-f no- mrno’fsjmc f-Ho -F-f ^c^ r»F P! Fortniil ation i g n'rpFo'rahl p 

but when examining the productions of overall objectives the alternative 
expression is more meaningful » The contribution of production of over- 
all objectives to E is shown in Table 7. Also presented in this table 
is relative productivity, REL(E*). 

ic 

Effectiveness figures, Ej^, did not convey any particular meaning 
to the decision-maker, due to the fact that he had no standa.-d by which 

to relate the findings. Two types of standards could have been utilized. 

* 

The first and most natural way to compare E'-s would be to compute the 
comparable figure for 1966-67. In Radnor's case, this was not possible 
since 1967-68 was the first year this kind of data were collected. 

A means of comparison would be to express the actual production fig«ires 
relative to what productivity would have been assuming perfect perfor- 
mance. These relative effectiveness values were computed to provide 
evaluative information to the Assistant Superintendent about the 
productivity of the overall health and physical education objectives. 

The Emergency Objective, Oq, was produced at nearly perfect levels 
of performance. Health Eduction, Oj,, as an overall system objective, 
had the largest deficiencies in terms of relative effectiveness. Over- 
all relative effectiveness was at the 0.81 level, indicating plenty of 
possibilities for increasing productivity. 

Table 8 examines the productivity of programs in a similar vein. 

The Emergency Program, as was expected, was the best on any productivity 
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criterion. It so happens that this program was the only producer of the 
overall objective of the same name. The Health Education Program, Pg, 
exhibited a poorer production record than the overall objective of the 
same name, indicating that health education activities incorporated into 
other programs were producing at an average level of relative effective- 
ness greater than the level in program Pg itself (0.57). This also 
pointed out the need for someone in the Radnor system to coordinate the 
health education efforts. 

Some of the problems in programs such as Annual Screening 
Communicable and Infectious Disease (P 3 ) could easily have been resolved. 
Some of the lower scores in Pj^ resulted from the follow-up procedures. 

For instance, the nurse was using expert judgment and receiving a low 
performance criterion score. Subsequent iterations of this model should 
be based on a more thorough discussion of criteria at several levels so 
as to be more realistic. As was pointed out earlier, detailed output by 
activity was less meaningful for the decision-maker in this study. These 
data are given in Appendix D. 

B. Overall Objective Effectiveness by Program 

A cross classification showing the production and overall objectives 
allowed the decision-maker to see which programs were directed toward 
which overall objectives and the relative effectiveness of these en- 
deavors. Table 9 shows the distributions. 

While the relative productivity of Appraisal (0^) was 0.81, it is 
clear that the Communicable and Infectious Disease Program (P 3 ) was 
a low contributor. 



77ihe Assistant Superintendent for Curriculum had indicated that 
this was planned for school year 1969-70. 



EFFECTIVENESS AND RELATIVE EF^KHTTVENESS BY OVERALL OBJECTIVE AMD PROGRAM 
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An inspection of Appendix D would indicate that certain appraisal 
activities in P3 had low performance ratings. This was due to the judg- 
ment that venereal disease. was not a problan for the Radnor school popu- 
lation, and, therefore, activities directed at this problem were minimal. 
Therefore, unless the dimensions of the problem change, there seems to 
be little reason to consider activity changes with regard to this aspect 
of 0^, 

The overall objective, Follow-Up (Og), was produced in six of the 
eight programs. Actually, the one activity in the Follvow-Up Program 
(P5) is a residual activity. Problems with follow-up appeared to be 
more critical in programs P^, P2 and Pg. The Annual Screening Program 
(P,) and the Physical Examination Program <P«) actually performed a 
great deal better than the scores indicated. This is based on a 
realization that record-keeping for purposes of running programs such 
as these could not be expected to meet criteria for systematic record 
keeping. It was concluded that the amount of effort required to modern- 
ize the health record system would not be worth the effort. One sugges- 
tion, however, was that more care should be exercised by physicians when 
completing forms foi pupils, and clear recommendations should be set 
forth when the doctors have something to specify about the care of the 
pupils , 

In Pg(the Adapted Physical Education Program) the follow-up was 
rated low due to a lack of information. The author suggested that if 
the program was worth doing, the benefits of the program should have 
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been demonstrated. 



78 



Comment on the Emergency Objective (Oq) has been given previously. 

While Health Education as an overall objective (0^^) was produced in 
six programs ) only the activities in the Adapted Physical Education 
Program (Pg) received a high score (0.90). The Assistant Superintendent 
indicated that someone would soon be brought in to assume coordination 
and development of these activities. One way to approach this problem 
would be to devise a "health education test" to determine how deficient 
the Radnor students actually are in health education. It is conceivable 
that children who are bombarded by many sources of information may 
know a great deal more than is generally supposed. At least a survey of 
this nature wjuld throw light on the Health Education needs of the Radnor 
student population. 

Relative productivity of the Physical Education Overall Objective 
was at level 0.82. Discussions with the Physical Education Coordinator 
indicated that this was not a bad score, especially in view of the 
activities involved. More will be said about this when costs are intro- 
duced. 



As was indicated before, the decision-maker in most circumstances 
should not be overly interested in detailed evaluation for activities. 



78 

This program was mandated by the State beginning with school year 
1967-68. While P^dnor did have some ongoing activities, there was some 
skepticism as to the merits of the program relative to those of other 
programs. The Radnor staff did generally want to give the program a 
trial, but it is interesting that the State can mandate a set of activi- 
ties such as these when in a particular setting a minimal nutrition level 
would be much more relevant. 
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In this vein it should be emphasized that no statistic has intrinsic 



worth. The utility or worth of a statistic is derived from two factors: 
1) the importance of the question it is designed to answer, and 2) the 
extcrt to which it answers the question. The statistics generated in 
this chapter are descriptive and are primarily of value in the specific 
Radnor context. The statistics that relate productivity to cost tend to 
have more general interest since the dollar is the frame of reference. 

C. Cost-Effectiveness Evaluation 
This section considers cost and effectiveness with the framework 
remaining ex post . Three types of outputs are considered here. First to 
be considered are some overall comparisons pertaining to programs. Then 
a measure called The Elasticity of Cost-Effectiveness is defined and some 
values are discussed. Lastly, the 1967-68 cost-effectiveness point is 
worked out by program. 



The effectiveness-cost average per $1,000 for the Radnor health and 



physical education programs was 9.2 in 1967-68. That is 

E* x$1000 = *t|^„x$1000 = 9.2 . 

/ J. 900, ilo 

c* 



Table 10 shows the same ratios by programs. Here the ratio is defined 
®m/ x$1000 

n-ie *1^ evident from an inspection of Table 10 that programs 



as 



m 



differed widely in amounts of production per unit of cost they produced 
during 1967-68. Production per dollar was highest in the Emergency Pro- 
gram (P^) and lowest in the two physical education programs (Py and Pg) . 
It is important to stress that these figures can only be put in perspec- 
tive by examining the respective effectiveness-cost curves. For example, 
consider the curves of Figures 10a and 10b. The left hand situation 
represents a case where the decision-maker has spent 10 units of resourcej 



o 
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for a production of 3 (P units) , while the right hand sketch 



P units 



P units 




2 4 6 8 10 



R units 

Figures 10a and 10b Cost-Effectiveness Curves 




indicates a production of 5 (P units) for 7 resource units. The respec- 
tive ratios are ^ and . It is clear that the decision to spend at 
particular levels in the future, ceteris paribus , depends on the incre- 
mental relationships between effectiveness and cost, and not the present- 
year ratios. Therefore, Table 10 w»:. not helpful with regard co 1968- 
69, but did give the superintendent a feeling for "where the system was". 

One way to evaluate past performance is to look at the system "with 
and without" the particular element being evaluated. This notion is 
suggested many times in the literature, especially in the literature of 
water resource development.®® This is as close to marginal analysis as 



^^This statement holds for ongoing program activities, but as was 
pointed out in discussions of Cases 8 and 9, project-type activities 
must be considered with regard to the remaining developmental costs and 
not merely incremental differences from the status quo. If the situation 
rests on a consideration of the activity before anything has been develop 
ed, then the full developmental cost is relevant when assessing project- 
type activities. 

®^For example, Otto Eckstein, Water Resource Development , (Cam- 
bridge: Harvard University Press, 1965), pp. 51-52. 
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can be expected when dealing with discrete activities and programs* The 
resulting comparisons resemble the elasticity measures that are abundant 
in economics. The Elasticity of Cost-Effectiveness is defined as 



(E 

cS/c*-c* 



where 






and 



r* = it r* o* 

“* -1 • 



For convenience let 

4 



E = 



E, 

iSP> 



and 



C* = 

^m 

c = c. 



The Elasticity of Cost-Effectiveness is defined as 



(E/C)„ 



4^yE-^ 




4^<E 

AP<C 



4 E(C-^) 

4C(E-4E) 

C 4 E - 49^ 



E^ ' iPtf' 
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Other elasticity measures, such as the Elasticity of Demand, assume 
that the factor ^Cfjj^ is of extremely small magnitude and drop it from 
consideration. In the present context, however, it is the very essence 
of the problem that cannot be taken as inconsequential. If the 

expression is separated differently, the results are more meaningful: 



The numerator is the ratio of the effectiveness to the cost of the 
element being evaluated and the denominator is the ratio of the rest of 
the system*s effectiveness to its cost. When the ratio of the ratios is 
unity, the program being evaluated does not alter the average effective- 
ness-cost rati-.j for the system. A ratio less than 1.0 reflects the 
evaluation of a program producing at a lower rate per dollar than the 
average of the other programs comprising the system. If the ratio is 
greater than 1.0, then this indicates that a program is being evaluated 
which has a higher rate of productivity per unit of input than the aver- 
age of the remaining system programs. 

Table 11 shows the results of the calculation of the Elasticity of 
Cost-Effectiveness. This measure is also useful in Chapter V as a means 
of examining the implications of various controlled changes. The wide 
disparity among the elasticity values indicates that program production 
gains could be realized by careful consideration of cost-effectiveness 
criteria. The two physical education programs (P^ and Pg) were conspic- 



(E/c) - (c~ap) . ^ 

a:(E-gE) ■ E-tf ’ ’"“Itiplication is 



reversed and 
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TABLE 10 

COST, EFFECTIVENESS AND EFFECTIVENESS -COST, BY PROGRAM 



Program 




Cost ($) 



Effectiveness Effectiveness 

Cost ’ 





jj* m x$ 1000 

^kl 





2,961 


50.9 


17.2 


^2 


3,385 


72.9 


21.5 


^3 


913 


30.8 


33.7 


P 


3,595 


156.6 


43.6 


4 

^5 


180 


5.0 


* 


^6 


3,000 


18.1 


S.O 


^7 


27,259 


110.2 


4.0 


^8 


11,825 


41.6 


3.5 



Total or Average $53,118 



486.1 



9.2 
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TABLE 11 

THE Elasticity of cost-effectiveness, by program 



program Elasticity of Cost-Effectiveness 




1.98 

2.59 

3.87 

6.55 

3.06 

0.65 

0.28 



0.33 
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uous when their production contributions per unit of input were compared 
to those of the remainder of the system. 

The last evaluative data from school year 1967-68 are the cost and 
effectiveness figures from Table 10. Ordinarily, if expenditures were 
available at the activity level, there would be one effectiveness-cost 
point for each activity. The eight points taken from 

Table 10 are, in general, . In the next chapter other points 

in the neighborhood of are found and an attempt is made to 

sketch that portion of the effectiveness-cost curve so that insights 
for budget allocations can be found. 

Before entering Chapter V, it is appropriate to consider a basic 
assumption employed throughout the last part of Chapter IV. The 
assumption is that effectiveness per dollar of cost is a suitable 
measure of efficiency. This is not an attempt to argue that it is the 
best measure, but that several factors substantiate its reasonableness 
as a measure for school system evaluation. First, it must be conceded 
thrst school inputs are often associated with intangible and incommen- 
surable outcomes. Second, the basis of micro -economic analysis is mar- 
ginal comnari&on. On the prod'^ction side this is usually a measure of 
cost per physical unit produced. The effectiveness concept used in 
this dissertation resulted from an attempt to cope with the problems of 
intangible and incommensurable outputs and, at the same time, allow 
system alternatives to be considered. The major and most vulnerable 
assumption deals with the underlying concept of educational value. Its 
vulnerability, however, rests with the measurement techniques and not 
with the conceptualization. When more adequate values can be generated 
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from the decision-maker’s preference function, better economic compari 
sons can be realized* The next chapter considers the subsequent 
period allocation problems and the sensitivity of model outputs to 
changes in the decisicn=maksr*s value function. 



V COST-EFFECTIVENESS ALLOCATIONS FOR 1968-1969 



Earlier in this study it was advocated that the purpose of evaluation 
is to provide bases for improvement. This chapter deals with the transi- 
tion from evaluating school system behavior, after the completion of 
the school year to immediate considerations for the next year.®^ The 
logic underlying these procedures is found in the relationship between 
Cases 8 and 9 of Chapter II, Case 9 developed the basis for evaluating 
ongoing school systems, and Case 9 outputs, based on year t evaluation, 
are those that are required as input elements for Case 8 allocations in 
year t+l. 

The plan of this chapter is to present data relevant to the effec- 
tiveness-cost curves for each program. A listing of incremental relation- 
ships between effectiveness and cost for segments of effectiveness-cost 
curves is assembled and analyzed under three budget strategies: 1) the 
best allocation strategy, 2) the suboptimization strategy, and 3) the 
”make-everybody-happy*’ strategy. The implications of these strategies 
are then discussed and sensitivity analysis is utilized to explore outcome 

implications for selected modifications of the decision-maker *s value 
82 

structure . 



Ol 

Realistically school budgets are not prepared by a single decision 
at the end of the school year. This does not lessen the need for improve- 
ments in the allocation process.lt does imply that barometers (performance) 
measures)are needed to predict the K^(^values for the end of school yeart. 

82sensitivity analysis has been suggested as a way to consider the 
responses of models to variations in assumptions and parameter values. 

By this means, feedback can be obtained as to the appropriateness of apply- 
ing the model to systems with parameter values other than those studied. 
Some of the authors suggesting this type of analysis are: 

Hamburg, op. cit . , p, 14. 

Teichroew, Daniel, An Introduction To Management Science , (New 
York: John Wiley & Sons, Inc., 1964), pp. 259-260. 
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A. Effectiveness-Cost Data For Programs 
The Nursing Coordinator and the Physical Education Coordinator 
were interviewed so that effectiveness-cost information could be obtained. 
The Assistant Superintendent was interviewed for information relevant to 
the Health Education Program. The data shown below indicate the type of 
information that was presented to the respondents in order to help them 
with their estimates. 



Communicable & Infectious 
Disease 


Maximum 

Production 


Production 


1 


Detect venereal disease 


7.2 


2.2 


30 


Detect tuberculosis 


7.2 


6.8 


95 


Follow-up to parents 


6.6 


6.6 


100 


to teachers 


5.3 


5.3 


100 


to phys. ed. 


5.5 


5.5 


100 


Health education s^iill-over 


13.2 


6.6 


50 


Overall average performance 






73 



In addition to the information on the activities in the program, 
the respondent x^as shown a graph like the one in Figure 11. The graph 
showed the data corresponding to the program's effectiveness-cost point 
from 1967-68. Then the graph was discussed and the respondent was told 
that she (he) would be asked some questions about how average performance 
would change as a result of changes in costs. The term "average perfor- 



Werage Performance; 




1967-68 Cost 



Figure 11. Basis for Effectiveness-Cost Discussions 



129 - 



than terms such as "production," "productivity," or"eff ectiveness." 

The respondent was asked, as an opening question, "what would happen 
to average performance in Program m if you were to have an additional 
$1000 available? Whatever the response, a series of follow-up questions 
were asked such as "and why would performance go up?", or "what would 
improve that allows performance to rise?", etc. The procedure was 
continued by discussing the addition of $2000, etc. Cuts in program 
resource levels were also treated in the same manner. 

The estimates that resulted from the interviews with the three Radnor 
respondents are found in Appendix E. All things considered, the respon- 
dents seemed comfortable in the neighborhood of the 1967'"68 effectiveness- 
cost point and less comfortable as the discussion moved along the cost- 
axis in either direction. Generally, there was a "crisis feeling" 
about the present level of resources. That is, respondents were very 
willing to discuss additions to the expenditure level, but reductions 
tended to elicit a rigid response. This pattern was probably due to one 
or both of the following reasons: 

1. Administrators generally tend to feel that they can*t get along 
with less resources. 

2. The Radnor respondents felt that programs were operating at levels 

of return where small reductions in expenditures would be accompan- 
ied by large decrements of effectiveness. 

B. Incremental Effectiveness-Cost Data 

The relationships and estimates of Appendix E were separated into 
their components. Discussion has been focused on relative effectiveness 
or average performance but these outputs have to be put into absolute 
terms. Table 12 shows the incremental relationships ranked in order of 
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gain per $1,000. 

These estimates were translated into the framework of Case 8. In 
general, the decision-maker will add resources to the program as long as 
the additions continue to promise the best productivity per dollar. It 
is important to bear in mind that the human costs should be considered. 
When a decision-maker is overly bound to cost-effectiveness recommenda- 
tions and therefore is not discriminating about when to initiate program 
changes, apparent gains can be swamped by ignoring human costs (H^). 

TABLE 12 



EFFECTIVENESS-COST ESTIMATES. BANKED IN ORDER OF q<^IN PER THOUSAND DOILARS 



Program 

and 

Increment 

(Decrement) 


Magnitude 

of 

Estimated 
Gain (Loss) 


Estimated Cost 
Associated with 
Estimated Gain 
(Loss) 


Estimated Gain (Loss) 
per 1000 Dollars 


1-2* 


8.4 


1.0 


8.4 


6-2 


7.0 


1.5 


4.7 


7-1 


4.0 


1.0 


4.0 


1-1 


3.5 


1.0 


3.5 


4-1 


3.2 


1.0 


3.2 


7-2* 


2.7 


1.0 


2.7 


3-1 


0.9 


0.5 


1.8 


8-3* 


5.2 


3.0 


1.7 


7-3 


4.0 


3.0 


1.4 


7-4* 


2.7 


2.0 


1.4 


2-1 


0.9 


1.0 


0.9 


2-2* 


0.9 


1.0 


0.9 


2-3* 


0.9 


1.0 


0.9 


3-2 


0.4 


0.5 


0.8 


8-2 


1.5 


2.0 


0.8 


6-1 


0.3 


0.5 


0.6 


8-i 


0.5 


1.0 


0.5 


8 


(5.7) 


1.0 


(5.7) 


7 


(9.4) 


1.0 


(9.4) 


7 


(10.8) 


1.0 


(10.8) 


7 


(13.5) 


1.0 


(13.5) 


4 


(8.1) 


0.5 


(16.2) 


2 


(28.3) 


1.0 


(28.3) 


3 


(19.4) 


0.5 


(38.8) 


4 


(51.8) 


0.5 


(103.6) 



N0T£:Inf>'8a«ie Instances the Increment stands alone and in others the Incre- 



tient is related to a prior Increment. To incorporate this into the anal- 
ysis the increments are numbered. For example 7-1 is the first increment 
for program 7. An asterisk indicates dependence on a prior increment. 
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Before examining budget strategies, some prior observations should 
be emphasized. Human nature would seem to dictate that when an adminis- 
trator is asked what would happen if his accustomed level of resources 
were reduced, he would respond that the operation must suffer drastically. 
This seems to be borne out by the large estimated losses in productivity 
associated with small reductions from 1967-68 level of expenditures. 

Also, the coordinators indicated that most of thepDograms were near a 
leveling-off part of the effectiveness-cost curve, since gains associated 
with increased resource levels were far less than the program average 
without the incremental expenditure. 

The data also suggested that the present allocation (1967-68) was 
to some extent inconsistent with the superintendent’s preferences. This 
would appear l:o be the price he must pay for insisting that some overall 



or state -mandated program could impose this type of production constraint. 

Two sketches showing the relationship between productivity and cost 
are given in Figure 12, The left hand diagram describes the relationship 
between total productivity and cost, while the right hand diagram indir 
cates the relationship between marginal productivity and cost, with 
both curves assuming an underlying diminishing marginal productivity. 

The total production curve rises to its maximum when cost is C 2 » with 
marginal production at C 2 being equal to zero. If production continues, 
the implication drawn from the marginal product curve is that of a 
lesser output at C>C 2 than at C 2 . 

An ongoing program may be near its maximum but it is difficult to 
tell if it has passed a maximum in cost-effectiveness analysis. All that 
can be observed is that the marginal productivity of proposed Incremental 



objective, 0. , must be produced 




•. A policy 



expenditures is small and decreasing. Generally a zero increase in 
effectiveness associated with a proposed increment of cost suggests that 




Figure 12. Cost -Productivity Relationships 



expansion of the effort be discontinued. 

In cost terms, the marginal cost curve intersects the average cost 
curve at the latter's minimum. This is no more than a warning point. 
Under idealized pure competition conditions the firm will produce past 
the point of minimum average costs as long as the marginal cost is less 
than the unit revenue . With regard to school systems, the value of a 
unit of output must have the same meaning to the consumer as the producer 
if an optimal arrangement is to be obtained. There are practical deter- 
rents to this since even dollars have different subjective values to 
different people as well as to the same person under different circum- 
stances. The "second best" wayy to make these comparisons would be to 
strive for equality of educational value per unit of cost across the 



®%idney Weintraub, Intermediate Price Theory, (Philadelphia: 
Chilton Books, 1964), pp. 45-48. 
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various outputs producod by the educational system. At worst, this would 
yield an optimal productivity within the educational enterprise, fciven _a 
particular set of values . 



C. Budget Strategies 

A budget is a plan showing how much money is intended for alloca- 
tion to the programs of c» system. This study does not attempt to allocate 
the 1968-69 Radnor health and physical education budget, which would be 
presumptuous for many reasons. The present section does make a supposi- 
tion about the budget level, and, given the particular budget level, 
the implications of various budget strategies are explored. Data of 
Table 12 ^re utilized as the three budget strategies are considered. 

These strategies are: 

1. The B(best) strategy asks the decision-maker to allocate a $6,000 
increase to programs by finding the increment promising the best 
return per dollar from the remaining available alternatives at 
any given point in the allocation sequence. 

2. The S (suboptimization) strategy asks the decision-maker to allocate 
an undetermined sum to the best available return per dollar in each 
program. This guarantees that every program will have a budgeted 
amount greater than its 1967-68 expenditure level. 

3. The H(happy) strategy asks the decision-maker to allocate as much 
as is requested by each program, irrespective of return per dollar. 

Each resulting allocation is assessed in terms of three criteria: 

1. The overall system relative effectiveness 

2. The overall system effectiveness-cost ratio 

3. The Elasticity of Cost-Effectiveness 

1. A $6000 Increase for Strategy B 

To apply the mandates of strategy B the decision-maker examines the 
estimated gain per $1000 rankings in Table 12. He will select those 
increments that promise the highest payoffs as long as they do not exceed 
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$6000. He does not cart? which particular programs receive the $6000 in- 
crease from the 1967-68 expenditure level. 

According to this strategy, the preferred allocation wculd be to 
select increments 1-1 and l-2*(for para -professional assistance in the 
Annual Screening Program), 6-2 (for a part-time health education 
coordinator in the Health Education program, bearing in mind that this 
increment can be purchased without prior increment, 6-1), 7-1 (for equip- 
ment in the Physical Education Program), and 4-1 (for para-professional 
help in the Emergency Program). Those increments would expend $5,500 and 
therefore he cannot afford 7-2*. If 7-2* is so discrete that it cannot 
be broken into $500 package then he must select 3-1 or decide to leave 
$500 unexpended. The assumption is that 3-1 is selected (for classroom 
education on communicable and infectious diseases). 

The overall health and physical education system’s relative 
effectiveness rises from 0.81 in 1967-68 to 0.86 under strategy B. The 
effcctiveness-cost ratio, however, would drop from 9.2 per $1000 in 
1967-68 to 8.8. If spending $6000 in the best possible way lowers the 
average efficiency then the marginal efficiency is below the average 
efficiency for the 1967-68 version of the system. The results are 
consistent with the assessment that the system has passed the point of 
minimum average cost per unit. Under strategy B the $6000 was allocated 
to Programs 1, 3,4, 6 and 7. If the (E/C)^^ scores are compared to 
those of 1967-68 it is apparent that the intended distribution would lower 
the scores relative to 1967-68 for all programs receiving additional 
money . The one incremental exception is the equipment purchase for the 
Physical Education Program. This also suggests that the programs not 
receiving increases would rise in relative efficiency with respect to the 
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dollar gainers. Elasticity of Cost-Effectiveness scores are found in 
Table 13 and lend substance to this contention.®^ 

2. Strategy S 

The suboptimization strategy asks the decision-maker to select the 
best increment from each program. He seeks efficiency, but only within 
programs, one at a time . A strategy such as this does gain in one respect 
since each program will participate in spending the budgetary increase. 

TABLE 13 

ELASTICITY OF COST-EFFECTIVENESS FOR 1967-68 AND STRATEGIES B AND S AND 

H, WITK'^SOME SUMMARY 

MEASURES 



Program 

Pm 


(E*/C*) 

1967-68 


(E/C)„ 
Strategy B 


(E/C)n 
Strategy S 


(E/C)„ 
Strategy H 


?1 


1.98 


1.51 


1.64 


1.87 


P2 


2.59 


2.70 


2.26 


1.76 


?3 


3.87 


2.75 


2.99 


2.51 


P4 


6.55 


5.29 


5.75 


C AO 
U • HO 


P5 


3.06 


3.19 


3.46 


3.91 


P6 


0.64 


0.62 


0.68 


0.69 


P7 


0.28 


0.32 


0.38 


0.37 


P8 


0.33 


0.35 


0.29 


0.32 


REL(E) 


0.81 


0.87 


0.88 


0.90 


E*/C*($000) 9.3 


8.8 


8.1 


7.2 



Although relative effectiveness for the health and physical educa- 
tion programs, overall, rises slightly to 0.88, the productivity per 
$1000 falls sharply to 8.1. Table 13 reveals the elasticity values 
for strategy B and comparable data from 1967-68, 




Elasticity calculations are shown in Appendix F. 
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In the instance of strategy S, two programs exhibit lower scores 

than strategy B and six show higher scores. This is due to the $5000 

intended for the Adapted Physical Education Program (Pg) . An expenditure 

of $5000 in that manner would be so inferior (in economic terms) that 

other programs would end up "looking good" in a relative sense. The key 

figure is the overall production per $1000, which indicates the price that 

85 

must be paid in order to suboptimize. 

3. Strategy H 

The strategy suggesting that the decision-maker should give to 
the programs whatever they need is poor in general and is disastrous 
when marginal productivity is less than average productivity. 

Overall relative effectiveness climbs to 0.90 but the rate of 
productivity per $1000 falls to 7.2 from 9.2 in 1967-68. The elasticity 
figures in Table 13 behave as would be expected. Programs receiving 
hypothetical allocations, above the $6000 allocated under strategy B, 
suffer according to the elasticity measure. 

4. The Three Strategies 

The three strategies, when compared, show what happens when economic 
considerations are ignored. The critical measure from an economic view- 
point is the overall ratio of production to cost. It seems that person- 
nel administering programs, if they are not asked to consider economic 
criteria, will tend to examine overall relative effectiveness alone and 
not strive for a balance between effectiveness and cost. 



fhe Follow-up Program (P 5 ) was not changed, yet its elasticity 
score rises as more money is put into the system for other programs. 
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The next section examines the sensitivity of model outputs to 
changes in the decision-maker's value structure. 

D. Sensitivity Analysis 

This section deals vith some changes in the preferences of the 
decision-maker. The purpose is to determine the response of sclented 
output m.easures to changes in the value structure which was used. It 
is apparent that changes in preferences could transfortn inefficient 
programs into efficient producers and vice-versa. Two value structure 
changes were studied. 

1. The Emergency Objective (Oc) was nearly perfect in terms of 
relative effectiveness and appeared to be high in effectiveness 
per dollar. The parameter V(Oc) was tripled to study the 
influence of a substantial magnification of the importance of an 
overall objective which reflected high system performance. 

2. The Physical Education Objective (Oj;) was a little above average 
in relative effectiveness but was low in efficiency according to 
elasticity measures. The parameter V(0e) was tripled to study 
the influence of a substantial magnification of the importance of 
an overall objective which reflected average system performance. 

Each of the newly generated value systems were examined in terms of: 

1. Overall system relative effectiveness 

2. Overall system effectiveness-cost ratio 

3. The elasticity of cost-effectiveness 

4. The allocation of $6000 under strategy B. 

a. The resulting overall system relative effectiveness 

b. The resulting overall system effectiveness-cost ratio 

c. The resulting elasticity of cost-effectiveness values, if 
necessary 



1. Tripling V(0(,) 



When the overall objective Oq is tripled in value, the contributions 
of all activities related to Oq are magnified. The resulting value 
structure is presented next, along with the 1967-68 base value set. 



er|c 



1 



- 138 - 



0^* ^ (V(0(j) ) 1967-68 





109.9 


109.9 




100.3 


100.3 




420.0 


140.0 




125.0 


125.0 


Oe 


125.1 


125.1 


TOTAL 


880.3 


600.3 



The procedure involved a distribution of the revised total value to 
the system activities. The distribution structure provided in Chapter 
III (Table 2) was carried over to this situation. In this instance 
three activities were involved s?nce only they contributed to 0^. These 
activities were in the Emergency Program, (P^), although this program 
also has three additional follow-up activities that were not involved in 
the value modification. The resulting productivity potentials, by 
program, were: 





.Sv(a^O^) 

69.6 

88.4 

45.0 
442.0 

5.3 

31.6 
134.9 

51.5 

12.0 



TOTAL 



880.3 
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Although perforaance scores were carried over from the 1967-68 case 
study, the value produced in changed, even though production scores 
for the other programs were held constant. 

P_ 

Vi 50.9 

P 2 72.9 

P 3 30.8 

P 4 425.9 

P 5.0 

5 

Pe 18.1 

P^ 114.1 

A 

TOTAL 760.3 



The overall system relative effectiveness rises to 0.86 from 0.81 
in school year 1967-68, solely as a result of the change in value struc - 
ture . The ratio of production per $1000 rises to 14.3, but this is 
largely a function of the arbitrary value change, A standard such as 14.3 
would be very useful , however, in assessing the effects of system changes 



given the new value structure. 



The Elasticity of Cost-Effectiveness is useful even though the 1967- 
68 context has been altered, since the ratio is interpreted apart from 



any particular school context. The resulting elasticity outcomes were: 



Pm 




(E/C) 


Pi 


1.22 


1.98 


P2 


1.56 


2.59 


P 3 


2.41 


3.87 


^4 


17.54 


6.55 


P5 


1.95 


3.06 


P6 


0.41 


0.64 


P 7 


0.17 


0.28 


pg 


0.20 


0.33 



(1967-68) 



o 
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These elasticities were based on the 1967-68 program expenditures. 

The 1967-68 elasticities are shown on the right as a basis for comparison. 
The elasticity shift is in the direction of and, of course, away from 
all other programs. 

On the assumption that the revised value structure with 3jj^V(Ogi| 
was the appropriate set of values and the decision-maker was faced with 
the allocation problem treated earlier in this chapter, the budget 
strategies can be re-examined. 

First the decision-maker considered how he would allocate the $6000 
increase over the 1967-68 expenditure level by applying the mandates of 
strategy B. Then he needed a revision of Table 12 that reflected the 
same performance-cost relationships but embraced the preference shift. 
Table 14 provides these revised values. It is interesting that the 
only realignment among the increments was that 4-1 jumped from sixth 
to first place. And were the hypothetical conditions imposed upon the 
decision-maker, the same purchases would have been made. Overall system 
relative effectiveness was elevated to the 0.90 level. Productivity per 
$1000 for the entire system of health and physical education programs 
would have been 13.5 as compared to 14.3 for school year 1967-68 (given 
the revised values for oc). 

This indicated that a rise in the valuation of a well-produced 
objective is not sufficient to elevate the overall system efficiency 
scores unless there is adequate opportunity for expenditure in the 
highly valued area. Since R£L(Eq) was already at the 0.97 level there 
was little room for expenditure on the activities contributing to Oq. 

The elasticity calculations are omitted in this instance since they 
convey little additional information. 
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TABLE 14 



EFFECTIVENESS-COST ESTIMATES, HANKED IN ORDER OF GAIN PER THOUSAND DOLLARS : 



3tv(Oc3 

Program 

and 

Increment 

(Decrement) 


CASE 

Magnl tude 
of 

Estimated 
Gain (Loss) 


Estimated Cost Associated 
With Estimated Gain (Loss) 


Estimated Gain(Loss) 
per 1000 Dollars 


4-1 


8.8 


1.0 


8.8 


1-2* 


8.4 


1.0 


8.4 


6-2 


7.0 


1.5 


4.7 


7-1 


4.0 


1.0 


4.0 


1-1 


3.5 


1.0 


3.5 


7-2* 


2.7 


1.0 


2.7 


3-1 


0.9 


0.5 


1.8 


8-3* 


5.2 


3.0 


1.7 


7-3 


4.0 


3.0 


1.4 


7-4* 


2.7 


2.0 


1.4 


2-1 


0.9 


1.0 


0.9 


2-2* 


0.9 


1.0 


0.9 


2-3* 


0.9 


1.0 


0.9 


32- 


0.4 


0.5 


0.8 


8-2 


1.5 


2.0 


0.8 


c-1 


0.3 


0.5 


0.6 


8-1 


0.5 


3.0 


0.5 


8 


(5.7) 


1.0 


(5.7) 


7 


(9.4) 


1.0 


(9.4) 


7 


(10.8) 


1.0 


(10.8) 


7 


(13.5) 


1.0 


(13.5) 


2 


^28.3) 


1.0 


(28.3) 


3 


(19.4) 


0.5 


(38.8) 


4 


(22.1) 


0.5 


(44.2) 


4 


(141.4) 


0.5 


(282.8) 



NOTE: In some instances the Increment stands alone and in othes the 

Increment is related to a prior Increment. To Incorporate this 
into the analysis the increments are numbered. For example, 7-1 
is the first Increment for program 7. An asterisk indicates 
dependence on a prior increment. 



2. Tripling V(Oj;) 

The procedures of this section are identical to those of the previous 
section. Here V(O^) was restored to its 1967-68 level and V(O^) was 
magnified threefold. The resulting preference function is given below 
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and Is accompanied by the 1967-68 values for purposes of comparison. 





V(o^) 1 3 [v((^3 


1967-68 


°A 


109.9 


109.9 




100.3 


100.3 


°c 


140.0 


140.0 




125.0 


125.0 




375.3 


125.1 


TOTAL 


850.5 


600.3 


When the 


new aggregate value 


total was dis 


the division was: 

Pm 

A -a 




Pi 


69.6 




Pj 


88.4 




P 3 


45.0 






162.0 




P 5 


5.3 




p& 


31.6 




V 


341.5 




^8 


95.1 




Px 


12.0 




TOTAL 


850.5 





The 1967-68 performance scores were applied to the revised value 
structure emphasizing 0 . The resulting productivity scores for programs 
were: 
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50.9 

72.9 
30.8 

156.6 

5.0 

18.1 

291.0 

74.2 



TOTAL 700.5 



Overall system relative effectiveness rose to 0.82 compared to the 
0.8i of 1967-68 when values were undistorted. This slight increase in 
overall relative effectiveness meant that the productivity-cost relation- 



ships in the two Physical education programs were a little better than 
the system average. The ratio of production per $1000 was 13.2 and, 
as was pointed out earlier, has little interpretive value in Isolatioii 
since it is largely a function of the magnitude of .le values. This 
ratio would be useful as a basis for comparing contemplated alternative 



actions given this particular frame of reference, i.e. assigmaent of 
values. 



The resulting elasticity measures were: 



^m 


(E/C),, 


(E/C) 




1.33 


1.22 


Po 


1.71 


1.56 


L 

?3 

P. 


2.62 


2.41 


3.97 


17.54 


Pc 


2.11 


1.95 


5 

P, 


0.44 


0.41 




0.67 


0.17 


P8 


0.41 


0,20 



L1967-6 



i 




A comparison of the two «ets of elasticity figures showed that the 
increase in valuation caused a favorable shift toward programs P.^ and Pg, 
which were oriented toward Og, while programs not producing 0^ declined 
substantially. 

This provided an opportunity to test the conclusion drawn from the 
case in which 0 was tripled. In that instance it was concluded that 
not only would an objective have to be magnified in value to influence 
allocations substantially, but also there would have to be available 
opportunities for expenditures in the over-values domain. In order to 
assess thi.s, a new allocation of the $6000 was considered by constructing 
a new table similar to Tables 12 and 14. Table 15 lists these estimates 
based on the relative performance -cost curves used earlier. 

When increments were ranked according to the values reflecting the 
shift toward Og, the higher rankings contained a great deal of P^, the 
Physical Education Program. The Adapted Physical Education Program, (Pg) , 
needed more than the push it received from tripling Og. But it should 
be pointed out that only one activity of the four in Pg was influenced 
by the revaluation. The new allocation based on strategy B would be 
$2000 to P^, $2000 to P^, and $1500 to Pg. The total of $5500 fell 
short of the $6000 available but not many alternatives were available. 
Increment 3-1 was far down the list as a result of the change in valuation 
and the next two increments in line being too costly. A compromise 
decision was made to allocate only $5500. 

The resulting overall health and physical education relative 
effectiveness score was 0.86, as opposed to 0.82 prior to the allocation 
of the $5500. The overall effectiveness per $1000 fell slightly to 12.6, 
versus 13.2 prior to the allocation. This indicates that the $5500 did 

not help average system efficiency. 
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TABLE 15 



Lvr riUixviLWiiiDj: 

Program 

and 

Increment 

(Decrement) 


) i CjO l xrm 

Magnitude 
of Estimated 
Gain (Loss) 


1 

3[v(0EiI CASE 

T7 ^ Am Am ^ 

u xlua ucu \j\jo u 

Associated with 
Estimated Gain 
(Loss) 


Estxmatsd Gain (Loss) 
Per 1000 Dollars 


7-1 


10.2 


1.0 


10.2 


1-2* 


8.4 


1.0 


8.4 


7-2* 


6.8 


1.0 


6.8 


6-2 


7.0 


1.5 


4.7 


1-1 


3.5 


1.0 


3.5 


7-3 


10.2 


3.0 


3.4 


7-4 


6.8 


2.0 


3.4 


4-1 


3.2 


1.0 


3.2 


8-3* 


9.5 


3.0 


3.2 


3-1 


0.9 


0.5 


1.8 


8-2 


2.8 


2.0 


1.4 


8-1 


1.0 


1.0 


1.0 


2-1 


0.9 


1.0 


0.9 


2-2* 


0.9 


1.0 


0.9 


2-3* 


0.9 


1.0 


0.9 


3-2 


0.4 


0.5 


0.8 


6-1 


0.3 


0.5 


0.6 


8 


(10.5) 


1.0 


(10.5) 


4 


(8.1) 


0.5 


(16.2) 


7 


(23.9) 


1.0 


(23.9) 


7 


(27.3) 


1.0 


(27.3) 


2 


(28.3) 


1.0 


(28.3) 


7 


(34.2) 


1.0 


(34.2) 


3 


(19.4) 


0.5 


(38.8) 


4 


(51.8) 


0.5 


(103.6) 



NOTE: 



increment is related to a prior increment. To incorporate this 
into the analysis the increments are numbered. For example, 7-1 
is the first increment for program 7. An asterisk indicates 
dependence on a prior increment. 



The resulting Elasticity of Cost-Effectiveness data were compared 
to the system prior to the $5500 allocation and the system under the 
1967-68 preference function(i.e. prior to changing V(0g). These compari- 



sons were: 
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(E/C) 

(After $5500) 


(E/C)m 

(Before $5500) 


(E/C)„ 

(1967-68) 




1.04 


1.33 


1.98 


^2 


1.84 


1.71 


2.59 


^3 


2.83 


0.63 


3.87 




4.27 


3.97 


6.55 


^5 


2.28 


2.11 


3.06 


^6 


0.44 


0.44 


0.65 


^7 


0.72 


0.67 


0.28 




0.46 


0.41 


0.33 


In all 


instances, programs 


that did not receive part of the $5500 


allocation 


(^2* ^3* ^4* ^5* ^8^ 


showed increased 


elasticity outputs 


relative to 


the new value set (i 


.e. 31 [v(0e)1 ). 


Programs 6 and 7 stayed 



roughly the same, indicating that the proposed increments would keep 
them close to their previous efficiency levels. Program 1 would suffer 
in terms of efficiency according to the allocation proposal. This is due, 
in part, to the fact that its products were devalued as a result of the 
increased valuation of production. Also the 11.9 gain would cost $2000, 
this is a 6.0 rate of gain per $1000 and is far below the average 
12.6 figure for all programs. 

Perhaps the most interesting observation involved the Physical 
Education Program, Py. Even with the tripling of potential value for 
most of i^s activities, the program is unable to absorb an increase in 
expenditure level in an efficient manner. It is true that (E/C)y rose 
to 0.72 from 0.28. But if the 0.72 is weighed against all other programs 
lumped together, the resulting comparison is 0.72 to 1.40 (which is the 
reciprocal of 0.72). 
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3. Sensitivity Analysis Conclusions 



The conclusions that can be drawn from the sensitivity analysis 



are: 



A. When an overall objective is overvalued (undervalued) relative 

to other overall objectives, it will affect relative productivity 
for the system as a whole but will not affect relative productiv- 
ity for activities. The direction in which the system’s overall 
relative productivity is pushed depends on the performance of 
the activities that are overvalued in relation to the average 
level of relative productivity for the system. 

B. When an overall objective is overvalued (undervalued) relative 
to other overall objectives, it will affect efficiency for the 
system as a whole. This is due to the fact that efficiency is 
a function of production and cost. Changes in preferences 
cause ctsnges in the values assigned to the outcomes produced 
by the activities. This, in turn, changes the aggregate value 
produced by the system and consequently efficiency is influenced. 
The analogy in the private sector would be an increase in price 
per unit of a given product altering efficiency relationships 
since it would then be efficient to produce more of the product, 
ceteris paribus . 

C. In order for a shift in valuation for an overall objective to 
influence the potential for allocating resources to the programs 
.producing that overall objective, there must be opportunity in 
these programs. Opportunity is restricted when the programs 
producing the overall objective are producing at very high levels 
of relative productivity and/or when the available outlets will 
produce at low levels of productivity per unit of input. 

D. Changes in the value structure are not sufficient to change the 
allocation pattern. Change'5 must be considered along with the 
resulting changes in output potential and the nature of the many 
production-cost relationships across the entirety of the system. 
When the relevant information has been supplied by each of the 
data files, the resulting allocation should be an improvement 

by virtue of being able to provide more educational production 
for the consumers of school services. 

E. Summary 

This chapter has taken the evaluation data of Chapter IV and used 



®^The “ ceteris paribus* ' is more acceptable if the price increase is 
based on an increase in consumer demand and not an arbitrary price in- 
crease by the producer. A price rise in response to a real increase in 
consumer demand should represent increased profit for the producer. If, 
however, the price rise is arbitrarily administered by the producer , there 
may be a corresponding response in consumer demand. Whether or not the 
producer gains depends on the nature of the consumer response. 
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it as a basis for entry into the problem of improving next year's 
allocation. By using evaluation of the present state of the system as 
a baseline there is a presupposition that the decision-maker needs to 
have a clear idea of \ihere ha is before he can proceed to where he wants 
to go. This feeling permeates the entire study and is best evidenced 
by the relationship between effectiveness and cost. In order to make 
projections of cost-effectiveness it is helpful to know where you 
presently are. In the absence of this information the situation degener- 
ates to an a priori decision structure ignorant of the system's immediate 
experiences. 

The section on budget strategies demonstrated the practical implica- 
tions to school administrators of alternative allocation strategies. 

The sensitivity analysis indicated that values are important in 
evaluating systems of educational programs but other factors such as cost 

and performance are also important. 

The last chapter provides a summary of the study, a set of recommen- 
dations to the Radnor Superintendent, and some suggestions for research 



related to this study. 



VI SUMMARY, CONCLUSIONS AND RECOMMENDATIONS 



This dissertation has focused on the problem of evaluation of the 
programs of ongoing school system!-. A theory, specifying general 
relationships among the variables required for evaluation, was developed. 
In addition, the necessary and sufficient conditions for appropriate 
application of cost-effectiveness analysis were specified. The model 
suggested that four basic information files were essential to this type 
of analysis. These information files include 1) a set of valued overall 
objectives which serve as a standardizing parameter set against which 
evaluations are made, 2) a fundamental structure relating system activi- 
ties to the overall objectives, 3) a set of performance criteria and a 
performance outcome for each criterion which determines the extent to 
which the activity produced what it was designed to produce, and 4) a 
set of activity expenditures. 

Based on these concepts a program evaluation model was applied to 
a sub-system of a school system. This case study commented on the prob- 
lems encountered in obtaining data for the infv.Luiation files required by 
the model to produce outputs for program evaluation based on educational 
and economic criteria. 

The remainder of this chapter discusses 1) the application of the 
model at Radnor, with soma recommendations for Radnor, 2) a general set 
of recommendations and conclusions, and 3) suggestions for further 
research. 

A test of the feasibility of obtaining the type of data required by 
the model was provided by the health and physical education programs in 
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the Senior High School of the Radnor Township School System. This was 
not without many practical probloms. Three sets of recommendations were 
made to the Assistant Superintendent for Curriculum (the decision-maker 



in the study, who is now Superintendent of Schools at Radnor 



\ 



1. Much of the evaluative output of Chapter IV has more interpretive 
value in relative rather than absolute contexts. It is possible 
to assess the relative product *.vity of the 1967-68 health and 



physical education systems with respect to each overall objective. 
It is also possible to assess relative productivity by program 
and activity within program. Clear standards are missing, however, 
since estimated or intended production from the previous period 
was not available. The first recommendation was to perform a 
1968-69 iteration of the model, if it would provide a satisfactory 
payoff in terms of questions that remained unanswered from the 
1967-68 study. 



2o In considering the 1968-69 allocation problems based on 1967-68 
evaluations and effectiveness-cost estimates, another recommenda 
tion seemed in order. Sometimes activities are conducted even 



though there are more efficient alternatives elsewhere in the 
system. For instance, it. may be preferred, with regard to 
economic criteria, to give a particular child an additional 
hour of social studies per week rather than an hour of music 
instruction. Yet a policy may exist which dictates that each 
child will receive two hours of music instruction per week. 
Under these circumstances the cost of music instruction to that 
child would be what he was unable to gain had he been given the 
social studies work. In this context it was recommended that 
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spending at levels higher than 1967-68 would possibly conflict 

with economic criteria given the stability of preferences, 

^ ^ 87 

performances, and costs. 

3. The next recommendation was based on the possible utility of a 
1968-69 iteration for the health and physical education programs. 

It appeared from the analysis of budget strategies and 1968-69 
budget possibilities that most of the questions were answered 
in this sub-system. It was recommended that this type of system- 
atic analysis be tried in other sectors of the Radnor School Sys- 
tem. Should this recommendation be adopted, the accounting and 
budgeting systems would require modifications so that intended 
and actual costs could be associated with activities. Also 
mentioned was the idea of pupil-need assessment. With detailed 
information about pupil needs it would be possible to direct 
activities to individual pupil needs. This would allow the 
school system to plan individual "prescriptions" so that each 
child could receive a program more relevant to his needs. 

The suggested expanded application of the model is for school year 
1969-70, so that adequate preparation can be made prior to the start of 
the school year. 

Broader implications for using this type of analysis derive from 

®^This is a suboptimization comment relative to one education system 
since alternatives outside this sub-system were not considered. It is 
possible, although not highly likely in the author's opinion, that the best 
returns per dollar spent are found in the health and physical education 
sub-system. Only a broader set of objectives, value assignments, and 
performance and cost outcomes can provide answers such as those generated 
by the model in this study. 
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Some problems encountered at Radnor. The principal probl&n is that 'rplic- 
it statements of objectives generally are not made in school systems. 

While some methods that can be directed toward defining objectives do 
exist, a rationale which permits school administrators to see the need 
for this does not appear to exist. The criteria for assessing methods 
for structuring objectives should be pragmatic. Do school administrators 
use them to structure objectives? Are the objectives generated useful 
in directing the system's activities? Would they be willing to use 
these methods even if they do not currently? 

Once these objectives are thought through, measurement problems 
arise. The measurement of subjective values is difficult for two reasons. 
First, measurement procedures need to be simple yet provide outputs of 
at least interval -scale quality. An appropriate level of compromise 
needs to be established so that more useful measures can be developed. 

The second factor inhibiting the measurement of subjective values is 
that people are usually ud' omfortable about sharing their values with 
others and often are afraid that explicitness will constrain them in 
the future. This is due in part to their unfamiliarity with "mathematical 
procedures" such as those used in this dissertation. 

Before this type of analysis can be used in other parts of school 
systems, there is a need for good performance criteria. Cognitive 
outcome criteria have been especially troublesome since it is difficult 
to avoid the problems of standardized testing. One way to approach this 
is to try for greater understanding of the relationships that link 
cost and performance, and performance and production of preferred outcomes. 
Increasing the quantity and quality of the communications between the 
decision-maker and those who engineer and operate his programs, probably 
would be helpful. Another source of assistance could come about by 
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better communication between quantitative researchers and those with less 
quantitative sophistication. This can provide a basis for improved 
measurement of critical variables. 

There are several other problems that invite future research. The 
measurement of subjective values is an area that should provide opportun- 
ity for much inquiry. 

Community values and preferences could be assessed to first find 
out what is wanted by generating value profiles and then determining 
the program mix that will produce the best allocation for each profile. 
This type of analysis, if handled properly, could foster a greater 
understanding of the community by school administrators and vice-versa. 

While statistical problems were treated in Chapter II, they were 
largely ignored in the application of the model to the Radnor school 
system. Future efforts embracing this approach should consider the 
distributions of the performance variables. This would introduce prior 
and posterior distributions and would constitute a natural lead-in to 
Bayesian evaluation and allocation models. 

The concept of the "value of sample information"^® as something to 
be considered prior to sampling, could be extended to the "value of a 
research proposal" or the value of an experimental design. With a 
carefully developed analysis it would be possible to weigh alternative 
proposals for a given school system by considering the expected contri- 
bution to the value structure. 

®®Schlaifer, Robert, Probability and Statistics for Business Deci- 
sions (New York; McGraw-Hill Book Company, Inc. 1959), pp. 443-456. 
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There is no doubt that the chief executive of a school system needs 
better ways to gather, process, and summarize information necessary for 
his budgetary decisions. One can visualize the superintendent of a 



large urban school system seated in front of a computer console late in 
the evening. He enters estimates of a preference structure, representing 
a vociferous critic group, into the keyboard. He also enters a budget 
constraint an'^ instructs the computer to allocate the new budget to 
programs in accord with the critic group’s preference structure. He 
waits for a real-time response from the computer. A great deal of 
developmental effort remains before this can become an operational 



reality. 
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TIig Eckstein-McKGan Ratio ArgutnGiit with Iinpllcations for Cost-Ef f GctivGiiGss 

THg usg of thG ratio of benefits to costs has been criticized on 
several grounds. The leading antagonist of the ratio is Roland McKean 
and the principal advocate is Otto Eckstein. Almost alj. arguments on 
this subject in the literature focus on the criterion (ratio or otherwise) 
as a means of ranking projects for future investment . 

The reason the controversy is important is that most practical 
applications of the methodologies embodying economic criteria use ratios 
as the criterion. Tliis is a natural consequence of the intangible, 
incommensurable nature of public sector benefits. McKean makes the 
following criticisms of the ratio approach: 

1. The ratio is undesirable because it might be misinterpreted by 

uncritical analysts. 

2. The ratio should not b#» used to compare "dissimilar*' investments. 

3. The ratio can be treacherous because it doesn't reflect the 
absolute magnitude of gains and costs. 

The first point is reasonable but it presupposes that benefit-cost 
ratios are misleading in all cases. The author suggests that when the 
ratio is utilized we should take extra care to caution users as to the 
potential dangers. 

The second comment by McKean is taken from the context of a dialogue 
with Eckstein. The latter has suggested that the ratio is appropriate 
for selecting preferred alternatives from projects that are "similar" in 

^McKean, op. ci t. , p. 110. 
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terms of capital intensity (turnover in McKean’s terms) and risk. IfcKean 
agrees that the ratio is appropriate in this type of situation but raises 
the question why not use the present value of net benefits since in this 
CBSB both critsria will point to the senie selections « Before snswerlng 
this question we can look at a part of Eckstein’s argument. 

Eckstein presents an analogy which considers a supermarket investment 

and a hydroelectric project investment. He shows that the benefit-cost 

ratio and the rate of return criteria give drastically different results 

and that each criterion tells little about the relative merits of the 

alternatives. This is because the supermarket represents a high risk 

venture whose success hinges on the chances that the location will 

assure profitability; while the hydroelectric proposition represents a 

large fixed investment over a long period of time and has a relatively 

known demand and minimal risk. In this latter case the rate of return 

criterion is sound but » in general, the criterion should be adapted to 

91 

the nature of the alternatives. 

Now we want to return to McKean’s question which asked why use the 
ratio if we have "similar" projects. The response is simple - often we 
face a situation where we are unable to assign dollar values to incommen- 
surable and intangible benefits and consequently are unable to obtain a 
net benefit determination. In 1965 the United Nations Research Institute 
for Social Development and Office of Social Affairs held a one week 
conference on Cost-Benefit Analysis of Social Projects. Norman Scott, in 



9 0ibid . 

^^tto Eckstein, Water Resource Development ( Cambridge: Harvard 
University Press, 1965X, p. 54-55. 
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sunnnarlzlng the discussions related to benefit-cost ratios, ssidj 

This srgutnent in favor of maxitnizing a ratio was defended on the 
grounds that social benefits and costs cannot be reduced to the 
same units a national scale nor, frequently, at the project 
level . 

McKean’s third criticism related to the very nature of ratios in 

that they conceal absolute magnitudes. One remedy is to indicate the 

absolute magnitude of numerator and denominator. But McKean himself 

provides another solution in another section of his writing. He indicates 

that benefit-cost ratios can be used to rank alternatives for a given 

budget. In such cases the analyst must be careful to exclude investment 

93 

costs not relating to the given budget. ^ 

The problem is that on one hand the ratio is dangerous on logical 
grounds but on the other hand it offers a way to express relationships 
when dealing with intangibles and incommensurables. In terms of cost- 
effectiveness analysi?;, ratios must be utilized. The reasons for this 
are the same reasons that give rise to the need for cost-effectiveness 
as an analytical tool. A final compromise is in order. When an analyst 
decides in favor of a ratio criterion he should be sure that 1) the 
absolute magnitude of the numerator and denominator are indicated, and 2) 
the same budget constrains the comparisons or, if different budgets are 
used, the emphasis is uniform with regard to the nature of the investment. 



^ ^Cost-Benefit Analysis of Social Projects , Report of a meeting of 
experts held in Rennes, France, 27 September-2 October 1965, United 
Nations Research Institute for Social Development, Report No. 7, p. 5. 



[cKean, op. cit. , p. 114. 
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APPENDIX B 

DESCRIPTION OF RADNOR HEALTH AND PHYSICAL EDUCATION PROGRAMS IN THE SENIOR 

HIGH SCHOOL 

1 . Atinug 1 Scr66tiing Program 

The target population is all Senior High School students. Ideally, 
the timing is as early as possible in the school year since the informa- 
tion is required for the Physical Examination Program ( for 11th grade 
students only) and for follow-up reasons. 

Each pupil is weighed and measured for height. Vision is tested and 
if there is uncertainty the child is retested in a few days. Hearing is 
tested for all 11th grade pupils as part of the physical examination, for 
new pupils who are not accompanied by adequate health records and for 
those pupils with records of hearing difficulty. The "Pupil Health 
Record" is updated and the nurse confers with the child and discusses 
problems and health practices. Pupils are referred to an appropriate 
agency when necessary. 

The following chart describes the flow of information and shows the 
interrelationship of the Annnual Screening Program to the Physical 
Examination Program, 
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2. Physical Examination Program 

The target population is all 11th grade pupils and new students who 
are not accompanied by adequate health records. The examination is 
given as early as possible in the school year to facilitate follow-up. 

In many cases private physicians conduct the physical examination and 
forward the results to the School Nurse by the first day of the school 
year. 

Each student has previously (unless he was examined by a private 
doctor prior to the beginning of the school year) been through the 
Annual Screening Program. Consequently, he has had vision, hearing, 
height, and weight checked. The child is checked for postural deviations, 
orthopedic defects, cardiac defects, etc. by the School Doctor. The 
"Pupil Health Record" is updated and the School Nurse discusses problems 
and health practices with the child. Students are referred to an 
appropriate agency when necessary. The chart describing the flow of 
information in connection with the Annual Screening Program is also 
appropriate here. ^ 

3 . Communicable and Infectious Disease Program 
The target population for this program is variable. With respect 
to communicable diseases, the program aims at all pupils. The tubercu- 
losis screening activity is cffared to 11th grade students. Actually, 
the State of Pennsylvania mandates a screening program for 1st and 9th 
grade children. The 11th grade children in 1967-68, however, had not 
been screened for tuberculosis in the 9th grade since the mandate went 
i.nto effect this year (piror to this 11th graders were tested). Conse- 
quently, this year's 11th graders and the 1968-69 11th grade pupils will 
be screened for tuberculosis in this program as well as the state-mandated 
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student population. 

Each child is givetk the Tine Test under direction of the School 
Doctor. Positive reactors are referred to Delai^are County Tuberculosis 
and Health Society Mobile X-ray Unit in the spring of the school year. 

In terms of other diseases teachers send pupils to the nurse and the 
nurse, after examination, recommends exclusion and medical care when 
appropriate. 

4. Emergency Program 

This program has a target population of all Senior High School 
students. 

The Emergency Call System provides a nurse at all times during the 
school day. This functions through the school secretary. In the event 
the nurse is not available a qualified back-up person is designated. 

Free ambulance service is ava..lable through the Radnor Police and Fire 
Departments. The school nurse must accompany the clilld to the hospital 
since no physician attends the ambulance. The family doctor is contacted 
as soon as possible. 

In instances of less serious emergency the nurse administers first 
aid. The nurse must evaluate the complaint and the symptoms observed. 

An est'^mate is made of the need for immediate or deferred medical care. 

In doubtful situations the family doctor or parent is consulted. Students 
are not sent home without the knowledge of the parent, an emergency person 
listed in the nurse’s files, or the family doctor, or, in rare cases, the 
school principal. Transportation is generally provided by the parent. 

Notations are made on the "Pupil Health Record" concerning all 
accidents and serious conditions. 



o 
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5. Follow-Up Program 

This program has a target population of selected students who have 
been referred. Since referral is a normal consequence of the Annual 
Screening Program, the Physical Examination Program, the Communicable 
Disease Program, and the Emergency Program., there is little left to be 
called ’’follow-up". 

The Follow-Up Program consists of those follow-up activities not 
otherwise classified. This amounts to the activities performed by the 
nurse to obtain a final disposition for each student referred. This is 
nsually done by telephone and personal contact with pupils but, in rare 
instances, the nurse could make a hom visit. Also, the Home and School 
Visitor may, in rare instances, make a home visit on behalf of the nurse. 

6. Health Education 

The target group for this program is all Senior High School children. 
As was mentioned earlier there are certain health education spillovers 
deriving from some of the other programs (e.g. Physical Examination 
Program) . 

A series of 17 Health Instructional Units were prepared on a pilot 
basis for 1967-68.^^ The classroom health education program was conducted 
in science, social studies and physical education. In addition, the 
nurse provides resource materials to the teachers. 

^Guidelines For Curric ulum Development in Health Education , Interim 
working manuscript prepared for Commonwealth of Pennsylvania, Department 
of Public Instruction, Bureau of General and Academic Education (November, 

1967). . . u .4: 

The Radnor Health Coordinator participated in the preparation or 

these pilot materials. 
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7. Physical Education 

This program has a target population of all Senior High School 
students. In fact, one of the program objectives is 100 per cent 
participation where this is meant to include each student on whatever 
basis he can participate. 

The program is designed so that there are three physical education 
periods par week for 10th and 11th grade students and two periods per 
week for seniors. There are games, sports, and activities. The sexes 
are segregated. A detailed curriculum of instruction governs the class- 
rooiTi activities. For example, in the 10th grade the instructional 
emphasis is placed on game fundamentals while in the later grades the 
emphasis is on scoring, rules and strategies. 

There is also a first-aid course given in the physical education 
curriculum. 

Varsity sports were not included as part of this study. Intra-mural 
activities operated on a very limited basis in 1967-68. 

8. Adapted Physical Education 

This program aims at pupils with postural and orthopedic defects 
and pupils with other physical deficiencies impairing normal physical 
participation in the Physical Education Program. 

This program supplements the regular Physical Education Program. 
Children are screened visually and then by photographic grid to select 
those with the greatest need for a special program. Those selected for 
the program are specially rostered. 

The program features special exercises and routines which are indi- 
vidualized to the needs of the child. Post-screening is also conducted 



at the end of the year. 
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APPENDIX C 



THE PERFORMANCE CRITERIA 



The general procedure to determine performance outcome scores was to 1) 
agree to a criterion, 2) determine how scores would be obtained, 3) 
obtain the outcome score, K*jO^*, and 4) permit the decision-maker to mod- 
ify the outcome score yielding 

The method used to allow the decision-maker the opportunity to change 
the raw score was to show him a sketch (see below and to discuss the 



problem with him in the following way: 

If we look at the diagram (Figure 13) we notice that the vertical 
axis represents value produced and the horizontal axis talks about 
performance. The line bisecting the angle is called K*V. This 
means that 80% performance is entitled to 80% of the value and 354 
of the performance received 35%. of the value. If you feel that the 
line K=V should be changed to reflect something else you^can do so. 
For instance, if you feel that a performance score of 80% is poor 
for a particular activity, you could have it produce a 70% value 
or, for that matter, any value you want. If you do change these 
relationships we will discuss their implications for your activities. 

The remainder of this appendix provides a brief discussion for 



each per^^ormance criterion within a program. 

1. Performance in the Annual Screening Program 
Vision appraisal (a^^ 0^) 

This objective was tested by an objective criterion. The total 
number of pupils to be tested was 934 and the total actually tes ed was 
905. ~ ■ 93 ? ==0*97 

The program specifications indicate that all Senior High School students 
should be tested annually. A few of the 29 "not tested" were actually 
tested during the second half of tht previous school year. The cut-off 
date for the purpose of analysis was arbitrarily set at 31 March 1967 




Figure 13. Value-Performance Discussion Graph 

(this comment applies to a-j^2 ^13 e<?ually). 

This Assistant Superintendent was presented with these data and a 

graph relating value to performance. Given the option of modifying 

A 

he chose to permit this to equal 
Hearing appraisal (ai20.) 

jn 

Hearing performance was assessed objectively. The hearing appraisal 
function was to be done for all 11 th grade pupils, those transferring to 
Radnor with inadequate hearing records, and those with prior hearing 
problems. The total to be tested was 378 and the actual number tested 
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was 3A3. 



K* 0, = ^ - 0.91 
12 A 378 



The decision-maker accepted “ ^12^A 



Height -weight ap praisal (ax3pA) 

This objective was also assessed by an objective standard. 

total to be tested was 934 with 918 tested. 

KtsOA = if = 0-98 % 30 ^ = K* 30 ^. 



The 



Discussion with Student for follow-up 

The pupil-nurse discussions in this program have two basic objectives. 

They are aimed at follow-up ai 40 g and health education 'Jhese 

objectives were both assessed subjectively by means of a semi-structured 
interview with the Nursing Coordinator. The format for this interview 
is described as follows: 

"1. One of the ideas built into the Annual Screening Program (we 

will treat the Physical Examination apart from this) is the dis- 
cussing of problems with the student with respect to follow-up . 

I will show you a scale and we will see if you can tell me the 
extent to which you feel the objective is presently (school 
year 1967-68) being met with a mind toward follow-up. We can 
discuss the scale to help you decide." 

The value she assigned to was 0.75 and the value for 

0.50. There were some qualifying remarks, 

This is difficult to estimate. If I had more time in the school 
year I think I could do a more effective job. Too much time is 
needed for paper work that can be done by a paraprofessional. 

The values for and a^^^Op were unchanged by the superintendent 

from the subjectivity assigned values. Figure 14 is a copy of the diagram 
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We do a perfect job 




We do Nothing 
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Figure 14. Typical Scale Presented to Discuss Performance Estimation 



presented for this type of situation. 

Referral of defects to parents (a]^50Q) 

Thi3 objective was assessed by an objective criterion. The total 
to be referred was 221 with 45 being referred, ® ® 0.20. 

There are several factors to be mentioned in connection with this 
statistic. Of the 60 defects for vision and hearing, 40 were referred. 

Of the 159 defects for height-weight imbalances, 4 were referred (two 
other defects were found as a spillover from this screening). This 
was a case of the Author and the Nursing Coordinator using different 
criteria. In addition, the 159 height-weight defects were based on a 
chart depicting average relationships with overweight and underweight 
ranges. This does not consider body structure. The nurse would not have 
had 159 defects by her more expert criteria.. In addition, there are 
many defects of which the nurse knows the parents are aware yet which 

the records may not indicate. She also may know when a student is on a 

, A 

special diet. Based on this information the Superintendent set K^50^= 
0.75. 

Referral of defects to teachers (aigOg) 

This is restricted to referral of defects to teachers other than 

physical education teachers. It was evaluated subjectively by the 

Nursing Coordinator in the semi -structured interview, and K*g0g * 0.80. 

Some qualifying remarks by the Coordinator were: 

I tell them in the grade-level meetings. If a child is really bad 
I go direct to the teachers. 

A 

was taken equal to K* 0^,. 

16 B lo 15 
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Referral of defects to physical education teachers (a 

This performance was assessed subjectively by means of a semi- 

structured interview with the Physical Education Coordinator. The 

following data were also presented to help form a judgment. 

To help you make a more informed judgment I can tell you that as 
a result of the Annual Screening during 1967-68 there were 37 
vision and 6 hearing defects referred to parents. There were 
159 height-weight defects but only 4 of these were referred. 

Based on these data and other Inputs the value assigned was * 0.40. 

Some remarks qualified this rating. 

We need specific written notifications since I can't always attend 
the grade-level meetings. The present method is not rsdequate.Also 
we nead quicker referral with cases pertaining to postural deviation. 

A 

The decision-maker let K*^0g * 

2. Performance in the Physical Examination Program . 

Reflex appraisal (a2i0^) 

This objective was assessed by an objective criterion. There is one 
problem connected with the objective measures for the Physical Examination 
Program. The Physical Examination is aimed at all 11th grade students 
and those transferring to Radnor without adequate physical examination 
data. The State of Pennsylvania allows for the examination of pupils by 
the private physician in cases where this is preferred by the parents. 
Those students not examined by the private doctor are examined during 
the school year by the school doctors. The reporting forms which both 
private and school doctors use are based on what can be called an 
"inference.' system". If the child is adequate or normal with respect to 
a given physical criterion the doctor leaves the space blank. Only where 
abnormality is manifest does the doctor make a mark or comment. When 
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someons reads the form it is possible to see a child's name and nothing 

else but the doctor's signature (sometimes not even the date). Therefore 

the ’’inference" is that the child is checked foi all defect possibilities 

during the examination. The total to be tested was 308 of which 4.11 were 

t-Afifpd K* 0 *= = 0.90. 

tested. 1^2 x'' A 308 

The Superintendent let ® 



Speech appraisal 

This was also checked objectively. The total to be tested was 308 

of which 277 were tested. ° “ *^22°*' 

Posture and orthopedic appraisal(a220^) 

This performance was assessed objectively with the total to be 
examined at 308 and the actual number of students tested at 277. 

K?30a = W = 



aeart and blood pressure appraisal (a ^^0^) 

This objective was assessed by objective standards. The number of 

pupils to be tested was 308 and the actual number tested was 280o 



Other appraisalCa^^O^) 

The criterion here was that if the examination was given "other 

things" were checked. ® 0.91, ^*5^A ” 

Referral of defects to parents (^26®B^ 

This performance was checked objectively. In all there were 2A 

defects to be referred (some pupils had more than one defect). Four were 

actually referred. KJgOg “ “ 0.17. 

The problem here is that the nurse's judgment as to what should be 

referred is superior to that resulting from a records check. In addition 
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parents often know of these problems and the nurse knows that they know* 

A 

The decision-maker set KJgOj “ 0.45. 

Referral of defects to teachers (a270g) 

This objective relates to referral of defects to teachers other 

than physical education teachers. It was evaluated subjectively by the 

Nursing Coordinator. * 1.00. 

Some remarks qualified the rating. 

I also tell the teachers about these in the grade - 
level meetings but I only refer serious defects - 
not minor details. If a child had athlete's feet 
I wouldn't tell his math teacher. 

A 

The decision-maker concurred with K^yOj » K^^O^. 

Referral of defects to physical education teachers (a2g0g) 

This objective was assessed subjectively by the Physical Education 

Coordinator. The following data were also presented to assist her 

judgment. "These figures represent the 24 defects referred to parents 

as a result of the Physical Examination. Included also are iwo defects 

that turned up as a result of the Annual Screening Program. There were 

5 reflex, 13 posture and orthopedic, 1 heart and blood pressure, and 7 

other defects." The criterion was assessed at K^gOg “ 0.45 with the comment: 

This is a little better reference to the referrals from the 
Annual Screening Programl. 

A 

The decision-maker allowed K* 0^ * K* 0_. 

28 B 28 ® 

Discussion of problems with student aiming at follow-up and education 
(*29®®^ » ^*29^^ 

This effort is aimed at follow-up ^ 29^8 health education 
The method of appraisal was a subjective rating obtained from the semi- 
structured Interview, K*g0g * 1.00, « 1.00. 

The decision-maker, however, felt that the resources available for 
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these efforts did not allow such a high performance and made the 

A A 

following adjustments: ® 0.80, 



3. Performance in the Communtcable and Infectious Disease Program 
This program operates at a more substantial level in the elementary 
schools, since pupil needs are greater there. 

Health education spillovers ( 3 ^ 201 )) 

Spillover benefits derive from pupil discussions with the nurse, 
in tuberculosis testing activities, and from literature readily available 
in the nurse *s office. The objective was evaluated subjectively by the 
Nursing Coordinator during the semi-structured interview. 



K* 0 
^32 D 



0.25. 



She had several qualifying remarks: 



We don't get too much in the way of communicable 
or even infectious diseases in the high school. 

I do supply resource materials to the teachers 
but if you mean does the school do what it can - 
no. There should be a specific health curriculum 
with a qualified teacher. Our effort is spotty. 



The Superintendent felt that the low rating was more a comment on 

the performance of the health education curriculum than the health 

education spillovers resulting from limited activities in the area of 

communicable and infectious diseases. Therefore the criterion was raised 

so that K* 0 * 0.50. 

32 D 

Veneral disease appraisa 1 ( 3330 ^) 

The evaluation of this objective is tricky. There is no specific 
activity directed toward this. On the other hand there does not appear 
to be a need to have an activity. The performance however was rated at 



0.30 since there are spillovers deriving from educational efforts aimed 



at detection awareness in the classroom. Also, the nurse pointed out 



« 172 - 



that the school doctor or private doctor would come across veneral disease 



cases if the condition was present. 



Tuberculosis appraisal (a 3/^0^) 

This activity was aimed at llth grade pupils during 1967-68. The 
activity is temporary and will only run in the Senior High School for 
one more year due to a change initiated by the State of Pannsylvania. 

It was at the initiative of the Nursing Coordinator that this group was 
tested, since the State’s change would have allowed the llth graders in 
1967-68 and 1968-69 to slip by untested. 



There were 325 pupils tested but only 308 eligible according to 
the numbering system used in this study. The decision-maker set 

%4°A ■ 

Refe?rrals of veneral disease and tuberculosis (a^^Og) , (a^gOg) , (a^yOg) 
This is referral to parents a350g, teachers other than physical 
education teachers a^^Og, and physical education teachers a^yOg. 
Interestingly enough, there were no defects found and the performance 
ratings assigned were all at 1.00. The utility of those ratings is 
questionable. 

4. Performance in the Emergency Program 



Care for serious injury (a4i0(,) 

This activity was assessed subjectively by the Nursing Coordinator 
during the semi -structured interview, “ 1*00. 

The qualifying remark was, "We handle them all." 

A 1 aa 

The decision-maker concurred with 0^ » 1.00 
Care for less serious injury(a240Q) 



The Nursing Coordinator's subjective assessment was also used to 
evaluate this performance. = 1.00. The decision-maker felt that this 
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A 

was somewhat high and set * 0»95<- 

Care for illness 

This was also assessed subjectively by the Nursing Coordinator during 

the semi-structured interview. K*,0^ * 1.00. The decision-maker adjusted 

A 

this value to “ 0.90. 

Referral and Follow-up of injuries and illness V^^Og) , (a^sOg) , (a^^OB) 

The referrals of serious Injury 344 ^ 2 , less serious injury 3 ^ 505 * 

and illness a,^0 were rated by the Nursing Coordinator during the semi- 
4b B 

structured interview. These were all rated 1.00 and the Assistant 

Superintendent > agreed. 

5. Performance in the Follow-up Program 
Ec "h program has some type of follow-up built into it. One activity 
which seemed to remain apart was the determination of the status of each 
student who was referred, with special emphasis on the Annual Screening 

and Physical Examination referrals. 

This was assessed objectively in that of the 54 referrals there were 

43 with definite depositions, K*^ 0 ^ = ^'3 * 0.80. 

5i ^ -'5‘Zr 

The decision-maker felt that this was extremely difficult to achieve 
and raised the performance value, ~ 0.95. 

6 . Performance in the Health Education Program 
This program is broken up into many small pieces due to the lack of 
a single person with responsibility and, of course, resources. Efforts 
are (and have been) underway for resolving some of the health education 

problems . 

As we have seen health education benefits are derived in a variety 
of wayj. A systematic, explicitly planned health curriculum, is preferred 
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at Radnor. With this in mind, the Nursing Coordinator wo.-ked during the 
summer of 1967 on a project to prepare a pilot curriculum. Instructional 
packages vere prepared for classroom use on a pilot basis during 1967-68 
but, as was previously mentioned, overall coordination was lacking. 

The Superintendent assessed the performance of the health education 

curriculum efforts in the Senior High School at 

According to the Radnor literature, the system also aims at the 
objective of instilling tie notion that each child must take the respon- 
sibility for his own health care. This activity, assessed by 

the Physical Education Coordinator at 0.75. While this assessment was 
subjective, it was based on data generated by means of a questionnaire 
given to 619 high school students. Table 16 presents the data and the 

question asked. The Superintendent let 

7, Performance in the Physica l Education Prcgraia 
In an effort to assess some of the physical education activities, 
the pupil questionnaire previously referred to was used. The interpre- 
tation of the derived data was left to the Physical Education Coordinator 
and her staff. She shared the data with her staff before rendering 
subjective assessments of the extent to which various objectives were met 
Par tic ipa t ion (ayi^) 

Built into the Radnor Physical Education Program is a belief that 
each child can and should participate in those activities his abilities 

allow. 

The Phyrical Education Coordinator assessed performance in this 
sense to be * 0.99. 

She remarked, ”We can't do much better in that we get them all now. 
I think we did miss one child for a while." 





TABLE 16 



STUDENT RESPONSES TO INDIVIDUAI, HEALTH RESPONSIBILITY 
QUESTION, BY GRADE AND SEX 



RESPONSE 


B 

10 


0 Y 
11 


S 

12 


G 

10 


I R L 
11 


S 

12 


Strongly agree 


42 


35 


62 


22 


48 


44 


Agree 


56 


32 


41 


70 


65 


65 


Disagree 


5 


4 


2 


2 


1 


3 


Strongly Disagree 


- 


2 


2 


- 


- 


1 


Dont* Know 


1 


3 


4 


2 


1 


3 


No Answer 


1 




• 


• 


1 




TOTAL 


105 


76 


111 


S6 


115 


116 



Do you feel that unless each student takes the responsibility 
for his (her) health, that his (her) health will suffer a 
great deal? 



QUESTION: 
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The decision-tnaker kept ^*^0^ * 

Fitness 

Fitness testzig is done by the physical education teachers. This 
testing has been done for years and the Physical Education Coordinator 
is extremely interested in these patterns. She subjectively assessed 
K* 20 g but qualified this figure. **We need more emphasis on this phase 
of our program.” 

The decision-maker amplified the need for greater emphasis on 

A 

fitness and lowered K* 0_ from 0.90 to K* 0„ * 0.80. 

72 E 72 E 

Physical confidence (ay 30 g) 

In order to assess the extent to which children have confidence in 
their abilities to engage in physical activities two questions were in- 
cluded in the pupil questionnaire. Table 17 shows the response data. 

The coordinator said: 

It would have helped on jowc questionnaire to have had another 
point on the scale between *very confident* and *not so confi- 
dent*. £a reference to the categories presented to the students 
on the questionnaire So what I am saying is that I think they 
should bs higher than the data indicated. 

A 

The decision-maker accepted “ ^73^E 0.80. 

Varie ty of skills and games (ay/Pg) 

The program description indicates that the programming was designed 

to expose the children to a wide variety of skills and games. This was 

subjectively assessed by the Physical Education Coordinator at K'^^Og— 1.00. 

She said: We do extremely well on the State -mandated programs 

in terms of the personnel available and the facili- 
ties available. Improvement is needed on the basis 
of our standards and goals as projected in terms of 
an interest-oriented program aimed at developing 
interest in lifetime sports. 

The superintendent felt that the performance should be assessed relative 
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TABLE 17 



STUDENT RESPONSES TO PHYSICAL CONFIDENCE QUESTION, BY GRADE AND SEX 





Girls -Grade 12 
Last Year 






Boys -Grade 12 
Last Year 


This Year 


More 1 


Same 


Less 


DK* 


Total 


More 


Same 


Less 


DK* ■ 


lotal 


Very Confident 


1 


47 


7 


1 


56 


1 


45 


21 


1 


68 


Not so Confident 


2 


36 


11 


1 


50 


4 


21 


12 


1 


38 


Not Confident 


- 


7 


1 


- 


8 


- 


1 


— — 


— 


1 


Completely net 






















Conf ident 


- 


1 


- 


- 


1 


— 


1 


— 




1 


Dont* Know 




- 


- 


1 


1 


- 


1 


1 


1 


3 


TOTAL 


3 


91 


18 


3 


116 


5 


68 


34 


3 


111 




Girls 


-Grade 11 








Boys -Grade 


11 




Last Year 










Last 


; Year 


This Year 


More 


Same 


Less 


DK* 


Total 


More 


Same 


Less 


DK* 


Total 


Very Confident 


1 


36 


8 


1 


46 


• 


22 


18 


1 


41 


Not so Confident 


5 


42 


11 


2 


60 


1 


11 


10 


3 


25 


Not Confident 




5 


2 


1 


8 


- 




2 


1 


4 


Completely not 






















Confident 


- 


- 


- 


- 


— 


— 


1 


— 


•• 


1 


Don’t Know 




- 


- 


2 


2 


- 


- 


— 


5 


5 


TOTAL 


6 


83 


21 


6 


116 


1 


35 


30 


10 


76 




Girls 


-Grade 10 








Boys -Grade 


! 10 




Last Year 










Last Year 


This Year 


More 


Same 


Less 


DK* 


Total 


More 


Same 


Less 


DK* 


Total 


Very Confident 


4 


33 


8 


1 


46 


3 


31 


20 


- 


54 


Not so Confident 


4 


27 


12 


- 


43 


3 


23 


18 


— 


44 


Not Confident 


- 


2 


2 


- 


4 


- 


2 


1 


— 


3 


Complete not 






















Confident 


- 


1 


- 


- 


1 


— 




•• 


• 




Don’t Know 




— 


- 


2 


2 


- 


- 


- 


4 


4 


TOTAL 


8 


63 


22 


3 


96 


6 


56 


39 


4 


105 



*Don' t Know 



QUESTION: How confident do you feel in your physical abilities (taking 

part in games, sports, and new physical tasks)? and what 
would your answer have been a year ago? 
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to the Radnor goals and not those of the State. 
Physical education attitudes (ay^Op) 



A 

He concluded K- 0 — 0.S5. 

74 E 



The physical education program attempts to develop positive attitudes 

toward the self; others, etc. The topic was directed to the students in 

the student questionnaire. Table 18 displays the response data. 

The Physical Education Coordinator rated this aspect of the progra: 

with K*_0 * 0.96. She added that the questionnaire waa more narrow than 

75 £ 

she would have liked: Physical education encompasses athletics in the 

true physical education sense of the word and on 
that basis we can use your data. 



The decision-maker allowed * K* 0 . 

75 E 75 E 



Lifetime sports (ay^Ojj) 

The physical education program took on an added emphasis on 1967-68. 
Built into the program was the idea that each child should be prepared to 
participate in some sport (s) on a lifetime basis. The true assessment of 
this goal should be made over an extended period of time. 

This was evaluated at K*g0p = 0.70. It was qualified by the Physical 
Education Coordinator. 

To a reasonable degree. (This was a reference to the lifetime sports 
activity). This is a projected aim and the implementation is based 
on personnel and facilities. It is highly desirable at Radnor. 

A 

The Assistant Superintendent lowered the rating to * 0.55. 



He added that progress seemed to be substantial but that there was a long 



veay to go. 

8. Performance in the Adapted Physical Education Program 
Posture and', orthopedic appraisal (Sgi^O^) 

While data were available on the number of children screened, it 
was difficult to assess the extent to which eligible children were 
appraised. It was decided to allow the Physical Education Coordinator 
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TABLE 18 



RESPONSE 


10 


BOY 

11 


S 

12 


GIRL 
10 11 


S 

12 


Very Successful 


48 


18 


43 


22 


45 


41 


Mildly Successful 


42 


42 


52 


57 


49 


53 


Not Successful 


7 


8 


12 


15 


22 


17 


Complete Not Successful 


6 


6 


1 


— 


— 


3 


Don* t Know 


2 


2 


2 


2 


— 


2 


No Answer 


— 


— 


1 


— 




-- 


TOTAL 


105 


76 


111 


96 


116 


116 



How successful do you feel physical education is in bring- 
ing out good attitudes in students? (such as, the importance 
of team play . fair play, how to be a good spectator) 



QUESTION: 
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to subjectively evaluate this activity, “ 1.00. 

She ooomented, ”We screened them all.” The decision”maker accepted 



^*8 A - 



Improvement in terms of defects (*32^3) 

The major part of the Adapted Program, in terms of pupil and teacher 
time expended, is aimed at correcting postural deviations through 



exercises and activities directed to individual problems. 

Some qualifying remarks were 

This is based on one year of diagnosis and work on 
exercises for deviations. It is not adequate to 
show strong improvement. A three-year basis would 
be a more reliable length of time. Referrals from 
medical exams could be better. 

The decision-maker assessed “ ^82^® " 0.®2, 

Attitudes (ag30jj) 

This program also hopes to improve the attitudes of the participating 
students. A question in the pupil questionnaire asked how well the 
student felt the program helped his desire to improve. Table 19 ahows 



the response data. 

The Physical Education Coordinator subjectively interpreted the 

data and rated the performance of this activity at « 0.90. Her 

qualifying remarks were: The creation of attitudes takes more chan a 

one-year stand. This was a new program - 
the orientation of participants, parents and 
faculty created a more favorable attitude. 

The Superintendent allowed ^*2®^ ■ 

Participation (a g^Og) 

Tue Physical Education staff at Radnor feel that the child could 
participate in more physical activities after realizing improvement in 
his physical state. A question on the pupil quest io.aiaire was directed 
toward this. Table 20 displays response data. 
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TABLE 19 

STUDENT RESPONSES TO ATTITUDES TOWARD IMPROVEMENT, BY GRADE AND SEX 



RESPONSE 



BOYS 

10 11 TOTAL 



GIRLS 
10 11 TOTAL 



Very much so 
Moderately so 
Not at all 

Don* t know 

TOTAL 



10 1 11 

15 13 28 

12 3 

2 13 

28 17 ^ 



5 16 

11 10 21 

8 5 13 

4-4 
28 16 44 



QUESTION: To what degree do you feel thst the adapted program created a 
desire for you to improve your physical status? 



12th grade pupils had already graduated when this questionnaire 
was given 



TABLE 20 

STUDENT RESPONSES TO ADAPTED PARTICIPATION QUESTION, BY GRADE AND SEX 



RESPONSE 


10 


BOY 

11 


S 

TOTAL 


GIRL 
10 11 


, S 

TOTAL 


About the same as before-B 


11 


24 


18 15 


33 


More confident in my 
ability 


14 


6 


20 


3 1 


4 


Less confident in my 
ability 
Don* t kn-'w 

TOTAL 


1 

28 


17 


1 

45 


1 

8 

28 16 


1 

6 

44 



QUESTION: As related to your participation in the adapted program, how 
well did it improve your desire to participate more fully in 
the physical education program? 



12th grade pupils had already graduated when this questionnaire 
was given. 



NOTE: 
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The Physical Education Coordinator assessed activity performance at 

K* 0 » 0.75 and made the following qualification: 

84 E 

Good plus - the ratio of participation is increased in relation 

.« . ^ ^ J /\r\ 'Th'I a “ts A fit^ff 

to cne scuaenc 8 unuer» tanuiug «uivi 
responsibility. 

A 

The Supterintendent let K* 0 ■ 



9 . Performance in the Dental Program 
Since the Dental Program did not operate in the Senior High School 
du ng 1967-68, the performance rating assigned to ^92^B 

0 . 00 . 
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APPENDIX D 

Supplementary Evaluation Outputs 

It was pointed out that a decision-maker with overall responsibili- 
ties is more interested in the system’s capabilities relative to overall 
objectives and programs, and, therefore, less interested in detail on 
activities. This comment can be appreciated better if one envisions 
the many such sub-systems for which a school administrator has responsi- 
bility. This appendix provides detail on productivity at the program 
activity level in Tables 21 and 22. 

The basic elements that form productivity outcomes are production 

outcomes at the activity level. These were shown in Table 6 and are 

repi^oduced in Table 21. In addition, it was seen earlier that relative 

performance at the activity level reduces to the performance score, i.e. 

K* 0*. Consequently Table 22 is a carry-wer from Table 4. 

mix 

Ordinarily, costs would also be carried over at the activity level 
and effectiveness-cost calculations could be given by activity. These 
figures would not be prescriptive for the coming year but would provide 
information as to the relative economic merits of the activities as they 
were conducted. These cost figures, as was indicated, are not available 



nor can it be expected that they will be available in most school systems. 
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0 



B 



TABLE 21 

VALUE PRODUCED BY ACTIVITY, PROGRAM AND OVERALL OBJECTIVE 

Emergency 



Annual 

Screening 



28.9 



Physical 

Examination 



Communicable 
ard Infectious 
Disease 





10.1 


^1 


8.6 


^34 


6.8 


4^2 


9.5 


(h.1 


8.6 


^33 


0.0 


^13 


9.3 


^3 


8.2 










^4 


9.5 










^5 


9.5 







44.4 



6.8 



^4 


4.1 


^6 


3.0 


^5 


6.6 


^44 


11.0 


^5 


3.3 


^7 


5.5 


■ ^6 


5.3 


^45 


5.5 




4.2 


^8 


2.5 


^37 


5.5 


®^46 


5.5 


^17 


2.2 




4.4 












13.8 




15.4 




17.4 




22.0 



0 . 



^1 65.9 

^2 39.1 

^3 29.6 



134.6 



% ^4 8.2 






13.1 



% 



6.6 



8.2 



13.1 



6.6 









Follow-up Health Ed- Physical 

• ucat ion Educat ion 






1 5.0 



Adapted 

Physical 

Education 



''81 9.1 



9.1 






2 1.9 
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Other 



A 

^91 0.0 



0.0 



ou 

92 0.0 



Total 



89.2 



5.0 



1.9 



0.0 



75.5 



134.6 



er|c 






62 



5.0 



11.8 

6.3 


^5 

^76 


15.2 

8.7 


18.1 




23.9 






24.5 




^72 


23.3 




^73 


21.0 




^74 


17.5 

86.3 


18.1 




110.2 



^4 



14.2 



16.4 



16.4 



41.6 



0.0 



84.1 



102.7 



486.1 
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TABLE 22 

ADJUSTED PERFORMANCE SCORES ^^0^* PROGRAM AND OVERAIX HEALTH AND PHY- 
SICAL EDUCATION OBJECTIVE 



OVERALL OBJECTIVES 



Program 


Appraisal 

(0*) 


Follow-up 

(Or) 


Emergency 

(Oc) 


Health 

Education 

(Op) 


Physical 

Education 


Annual 


11*0.97 


14*0.75 




14=0.50 




Screenin;_, 


12=0.91 


15=0.75 








(Pi) 


13*0.98 


16=0.80 












17=0.40 








Physical 


21*0.90 


26=0.45 








Examination 


22*0.90 


27=1.00 








iW 


23*0.90 


28=0.45 










24*0.91 


29=0.80 




29=0.80 






25*0.91 










Coinmunicable 


33*0.30 


35=1.00 




32*0.50 




and Infec- 


34*0.95 


36=1.00 








tious Disease 




37=1.00 








(P 3 ) 




44=1.00 


41=1.00 






Emerge ticy 




45=1.00 


42=0.95 






(?4) 




46=1.00 


43=0.90 






Follow-up 




51=0.95 








Health 












Education 








61=0. 75 




(Pe) 








62=0.40 




Physical 








75*0.96 


71=0.99 


Education 








76=0.65 


72=0.80 












73=0.80 


x-*' y / 










74*0.85 


Adapted 












Physical 












Education (Pg) 8 1*1. 00 


82=0.40 




83=0.90 


84=0.75 


Dental (?x) 


91*0.00 


92*0.00 
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APPENDIX E 

Cost-Effectiveness Data by Program 

The infomation in this appendix was provided by the two Radnor 
coordinators. The basic consideration was to ask them what would happen 
to performance, in each program, if the level of expenditure was varied 
in the neighborhood of the 1967-68 level of expenditure. 

1. Annual Screening Program 

The Nursing Coordinator felt that paperwork was the biggest handicap 
for improved performance in this program. She indicated that more could 
be done in the vay of follow-up and health education if the paperwork 
could be reduced. In her opinion an increase in resources should be 
devoted to part-time help so as to free the nurse for professional duties. 

When the Nursing Coordinator was questioned about decreasing the 
level of expenditure for this program, she indicated that the program 
was state -mandated and could not be cut. 

It should be emphasized that while only $2,961 was tied to this 
program in 1967-68, a substantial number of physical examinations were 
conducted by private doctors. The policy at Radnor is to (?;ucourage 
patronage of private doctors. This policy would not work well, for 
example, in the Philadelphia School System since family income for many 
pupils is low. Therefore, the cost per pupil for the same program would 
probably be higher since the school system would be forced to hire more 
doctors. 

The Nursing Coordinator's estimates of' relative effectiveness 



responses to selected cost investments were: 



Cos t ($ ) Re la t ive 

Effectiveness 



2,500 Not Rated 

1967-68 level - 3,000 0.75 

4.000 0.80 

5.000 0.92 



2. The Physical Examination Program 

In considering the Physical Examination Program, the Nursing 
Coordinator felt that increases in program performance could be realized 
by adding medical doctor resources to the program. The disposition of 
cases would speed up and bring about better performance scores on follow- 
up efforts. As in the case of the Annual Screening Program, the comment 
on private physicians also applies here. 

Small reductions in expenditures from the 1967-68 level would result 
in serious losses in relative effectiveness, according to the Coordinator 
while increases, in resources would not contribute very much. 

The Nursing Coordinator* s estimates of relative effectiveness respon 
ses to selected cost investments were: 



Cost($) Relative 

Effectiveness 



2,000 Not Rated 

2,400 0.50 

1967-68 level - 3,400 0.82 

4.400 0.83 

5.400 0.84 

6.400 0.85 



3. The Communicable and Infectious Disease Program 
The Nursing Coordinator indicated that the diseases included in this 
program did not usually present serious problems in the Senior High 
School at Radnor. She did point out that the nature of the problem was 
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different in lower grades. To communicate the nature of the high school 
problem she cited the Tuberculosis screening effort for school 1967-68. 
The effort yielded no positive reactors. 

Her feeling was that it would be fruitful to provide classroom 
instruction on communicable and infectious diseases. When questioned 
about a $500 reduction from the 1967-68 expenditure level, she remarked 
that this would be sufficient to "wipe out the program.” The estimates 
of the cost-effectiveness relationships were: 



Cost($) Relative 
Effectiveness 



400 0.30 

1967-68 level - 900 0.73 

1,400 0.75 

1,900 0.76 



4. The Emergency Program 

The Nursing Coordinator had responsibility for the Emergency Program. 
She felt that additional money could Increase effectiveness by freeing 
the nurse from paper work. 

A $500 cut from the present level of expenditures would have to be 



absorbed by medical supplies or the time spent by the nurse in the program. 



The nurse indicated that either of these alternatives would reduce rela- 



tive effectiveness to a 0.65 level. Interestingly enough, a $1,000 



reduction would not do very much more damage to the program. 



The estimates of cost-effectiveness relationships provided by the 



nurse were: 



Cost($) 


Relative 

Effectiveness 


2,600 


0.60 


3,100 


0.65 


3,600 


0.97 


4,600 


0.99 



1967-68 level - 
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5. The Follow-up Program 

The nurse's comments here were related to the need for paraprofes- 
sional assistance to reduce the burden of paper work. As a result the 
nurse would be able to spend more time on the telephone for follow-up 
purposes. The Author did not discuss dollar resources with the Nursing 
Coordinator in connection with this effort. (The 1967-68 cost-effective- 
ness point was described by a $180 expenditure and a 0.9A index of 
relative effectiveness.) 

6. The Health Education Program 

Since no administrator at Radnor had coordinated health education 
matters, the Assistant Superintendent accepted this responsibility. He 
indicated that by 1970 the Health Education Program could be expected 
to be more effective. This change would come about when a health 
education lecture room would be made available and when a person would be 
assigned the coordination of these functions. 

A slight increase in activity level could be brought about in 1968- 
69 by providing film strips and other educational materials for teachers 
to use. The Assistant Superintendent for Curriculum's cost-effectiveness 



estimates were: Cost($) Relative 

Effectiveness 



2,000 0.50 

1967-68 level - 3,000 0.57 

3,500 0.58 

5,000 0.80 



7. The Physical Education Program 

The Physical Education Program accounted for approximately one half 
of the 1967-68 total expenditure of $53,000. The Physical Education 
Coordinator indicated that there was competition between this program 




i 
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and the newly instituted Adapted Physical Education Program. The 
competitiveness centered on the need for equipment. Other important 
needs were physical facilities and physical education personnel. The 
possibility of facilities being increased is remote, but personnel are 



needed to lower class size so that 5-ncreased emphasis can be given to 
individual sports and individual needs ^ 

Increases of $1,000 and $2,000 should be accompanied by increases 
in effectiveness in the area of fitness since these resources would be 
directed to the purchase of equipment. Pupil confidence could also be 
expected to improve. 

A $5,000 increase would go toward hiring an additional physical 
education teacher as would a $7,000 addition. These resources would be 
devoted toward program performance aimed at lifetime sports. 

A drop of $1,000 would reduce effectiveness substantially. The 
largest losses could be expected in the lifetime sports activity and the 
variety of offerings activity. Still greater decreases in the level of 

resources would ’’push effectiveness way down . 

The Physical Education Coordinator's cost-effectiveness estimates 



were: 



Cost($) Relative 

Ef f ec tiveness 



24.300 

25.300 

26.300 

27.300 

28.300 

29.300 

32.300 

34.300 



0.60 

0.70 

0.78 

0.85 

0.88 

0.90 

0.93 

0.95 



1967-68 level - 
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7. The Adapted Physical Education Program 

The Physical Education Coordinator indicated that Improvement for 
this program is tied to three factors - space, personnel and equipment 
(this comment is similar to the Physical Education Program discussion). 

In response to the question ”what would be helped if you add $1,000^' 
she replied that the money could be used for more equipment. As the 
amount of available funds Increased she would apply the expenditures to 
personnel. 

With a large increase to $17,800 she could purchase equipment and 
hire 70% of a new person. Her estimate was that program effectiveness 
would rise to about 0.95. 

The 'Coordinator felt strongly that the program would experience 
decreases in effectiveness across all objectives if expenditure.'} were 
reduced. 

Cost-effectiveness estimates from the Physical Education Coordinator 



were; Cost($) Relative 

Eff ect ivenesf ; 



10,800 0.70 

11,800 0.81 

1967-68 level - 12,800 0.82 

14.800 0.85 

17.800 0.95 





X 
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APPENDIX F 

ELASTICITY OF COST-EFFECTIVENESS CALCULATIONS 

This appendix provides the data and calculations underlying the 
seven sets of Elasticity of Cost - Effectiveness outputs. 

1. The 1967-68 basic outputs 

2. The addition of $6,000 to 1967-68 funds, allocated by strategy 

I 

I 

B and resulting outputs 

3. Strategy S outputs 

4. Strategy H outputs 

5. The triple V(0 q) case and resulting outputs under 1967-68 outcomes 

6. The triple V(0g) case and resulting outputs under 1967-68 outcomes 

7. The triple V(0 e) case with $5,500 allocated by strategy B and 

resulting outputs 

The seven output sets are presorted on the next seven pages. Each 
set is referenced to the page of the text first showing the results. 



Output Set 
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